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22st  Annual  National  Test  &  Evaluation  Conference 
Jacksonville,  FL 
6-9  March  2006 


On-Site  Agenda 
Final  Attendee  Roster 


Monday.  6  March  2006 


Tutorials: 

How  to  Improve  the  Effectiveness,  Efficiency,  and  Integration  of  Test  &  Evaluation  (T&E)  and  Modeling  and  Simulation  (M&S),  by  Dr.  Mark  J.  Kiemele,  Air 
Academy  Associates 

Putting  the  Missions  and  Means  Framework  (MMF)  to  Work  to  Drive  Analysis  -  Getting  to  the  "So  What"  by  Asking  the  Right  Questions  about  the  Mission  and 
Generating  Complete  and  Relevant  Answers,  by  LTC  Britt  Bray,  USA(Ret),  Dynamics  Research  Corporation,  Inc.  and  Mr.  Bill  Yeakel,  President,  ORSA 
Corporation 


Tuesday.  8  March  2005 


Modeling,  Simulation  and  Testing:  Collision  Course  or  Happy  Marriage?,  by  Hon  Philip  E.  Coyle,  III,  former  Director,  Operational  Test  &  Evaluation  (DOT&E), 
OSD 

Panel:Non-Technical  Obstacles  to  Technical  Obstacles  to  Effective  M&S  Connectivity  in  Effective  M&S  Connectivity  in  Support  of  T&E: 

Panel  Chair:  Mr.  John  Illgen,  Northrop  Grumman  Simulation  Technologies 
Panel:DT&E’s  Perspectives  -  “Making  M&S  and  T&E  Better  Partners”: 

Panel  Chair: 

-  Mr.  Chris  DiPetto,  Special  Assistant,  Systems  Engineering,  Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  Technologies 

Panelist  (Government  Representatives): 

-  Dr.  George  Kailiwai,  III,  AFFTC  Technical  Advisor 

-  Mr.  Thomas  Berard,  Director,  White  Sands  Missile  Range 

Panelist  (Industry  Representatives): 

-  Mr.  Steven  E.  Cameron,  Senior  Manager,  Modeling  &  Simulation  Infrastructure,  Boeing  Analysis,  Modeling,  Simulation  &  Experimentation 

-  Mr.  James  A.  Hazlett,  Director,  Mission  Profiling,  Mission  Innovation,  Raytheon  Integrated  Defense  Systems 

An  LFT&E  Perspective  on  Making  M&S  and  T&E  Better  Partners,  by  Dr.  Lowell  Tonnessen,  Institute  for  Defense  Analyses 

Modeling  the  FCS  Brigade  Combat  Team  in  a  Stability  and  Reconstruction  Operations 
Mission  -  Implications  for  T&E  Planning,  by  LTC  (R)  Britt  E.  Bray,  Dynamics  Research  Corp. 

T&E  Transformation  -  "We  Don’t  Need  a  More  Formal  Invitation",  by  Mr.  William  P.  Yeakel,  President,  ORSA  Corporation 


Wednesday.  8  March  2006 


DMSO  Panel 

Panelist: 

-  Challenges  For  M&S  in  Army  Test  and  Evaluation,  Mr.  Roe  Mirabelle,  Chief,  Technical  Support  Division  (TSD),  Modeling  &  Simulation 


Military  Services  Perspectives: 

COMOPTEVFOR  Perspectives  on  Modeling  &  Simulation  in  Operational  Test  &  Evaluation,  RADM  Bill  McCarthy,  USN,  COMOPTEVFOR 
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The  Air  Force’s  Perspective,  Mr.  Jack  Manclark,  AF/Test  &  Evaluation  Directorate 
M&S  Support  to  FCS 


M&S  Studies  in  the  Context  of  the  T&E  and  Acquisition: 


Diagnosis  of  the  Problem:  What  Prior  Studies  Have  to  Say,  by  Dr.  James  E.  Coolahan,  Program  Manager  and 
Modeling  and  Simulation  Group  Supervisor,  National  Security  Analysis  Department 

"Open  Air  Testing  vs.  Modeling  and  Simulation:  At  last  Peaceful  Coexistence?",  by  RADM  Charles  "Bert"  Johnston,  USN  (Ret),  Vice  President  &  General  Manager, 
Wyle  Laboratories 

Live- Virtual-Constructive  Distributed  Environments  in  Support  of  Test  and  Evaluation,  by  Major  Kelly  A.  Greene,  Ph.D.,  Air  Force  Agency  for  Modeling  and 
Simulation 

Open  Technology  Development  &  Testing,  by  Mr,  John  Scott,  OSD,  AS&C  Consultant 

Modeling  and  Simulation  of  System  of  Systems  The  History  of  the  LPD  17  PRA  Testbed,  by  Mr.  Vincent  M.  Ortiz,  AVW  Technologies 


Thursday.  9  March  2006 


Using  the  Missions  and  Means  Framework  to  Generate  Cost-Effective  Live -Fire  Test  and  Evaluation  Strategies,  by  Dr.  Martha  K.  Nelson,  Franklin  &  Marshall 
College,  Department  of  Business,  Organizations,  and  Society  &  Mr.  Dennis  C.  Bely,  US  Army  Research  Laboratory,  Survivability/Lethality,  Analysis  Directorate 

Modeling  and  Simulation  to  Advance  Test  and  Evaluation,  by  Dr.  Sheldon  Tieszen  &  Dr.  Lou  Gritzo,  Sandia  National  Laboratories 

Track  1: 

T&E  / M&S  Architecture: 

-  Test  and  Training  Enabling  Architecture  (TENA)  Offers  Range  Interoperability  and  Resource  Reuse  Solutions  for  Test  and  Training  Ranges,  by  Mr.  Gene  Hudgins, 
TENA  User  Support  Lead 

-  A  Case  Study  of  T&E  Data  Supporting  A  Simulation  by  Mr.  Vincent  M.  Ortiz,  AVW  Technologies 
Policy  Proposals'. 

-  27  Steps  to  Success:  The  DoD  Acquisition  M&S  Master  Plan,  by  Mr.  Fred  Myers,  OUSD  (AT&L)  DS/SE,  Chair,  Acquisition  M&S  Working  Group 

-  “No  More  Second  Opinions:  Organizing  DoD  M&S  Within  the  Acquisition  Cycle”  by  Mr.  James  F.  O’Bryon,  The  O’Bryon  Group 

-  Optimizing  Cost  &  Time  Effectiveness  of  Test  &  Evaluation  Using  Knowledge-Based 
Simulations,  by  Dr.  Keith  S.  Bradley,  Lawrence  Livermore  National  Laboratory 

Track  2: 


(WA)  "V,  V&A  ”  Issues  to  Make  T&E/M&S  Work  Effectively: 

-  LPD  17  PRA  Testbed  VV&A  Database:  A  Disciplined  Approach  for  VV&A,  by  Mr.  Vincent  M.  Ortiz,  AVW  Technologies 
Examples  of  Attempts  to  Get  Mutual  Support  Between  T&E  and  M&S: 

-  Dual  Thrust  Modified  Smokey  Sam  for  Low  Cost  Testing  and  Simulation,  by  Mr.  William  R.  Taylor,  Tech  Advisor  for  EOCM,  AFRL/SNJW 

-  Integration  of  Transport  Modeling  with  Test  and  Evaluation  in  CBR  Building  Protection  Programs,  by  Mr.  James  Risser,  Battelle 


M&S  Studies  in  the  Context  of  the  T&E  and  Acquisition: 

-  CAW  use  of  M&S  in  Support  of  Maritime  HLD/HLS/ATFP  Test  and  Training  Exercises,  by  Mr.  Arman  Tchoubineh,  NAWC  WD,  Pt  Mugu,  CA 

-  Commericial  Air  Services:  Live  Test  Assets  for  Electronic  Warfare,  by  Mr.  Stan  Ulkoski,  PMA-207D2  Aircraft  Operation  Manager,  J.F.  Taylor,  Inc.,  NAVAIR 
Support  Contractor 
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“Improving  the  Test  and  Evaluation  of  Defense  and  Homeland  Security 
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HJSTRY&  TECHNOLOGY 


STRENGTH  THR( 


Personal  Note  to  Conference  Participants  from  the  T&E  Executive  Committee: 

Thank  you  for  attending  our  conference  in  Jacksonville.  We  appreciate  your 
making  this  meeting  a  priority  and  for  you're  taking  the  time  to  travel  to  be 
with  us.  Whether  you're  a  speaker,  exhibitor  or  simply  a  delegate,  we  hope 
that  you'll  feel  welcomed  and  that  you'll  fully  participate  in  all  of  the  activities 
planned.  If  you  come  with  friends  or  family,  there  are  a  couple  of  events 
scheduled  which  they  are  also  welcome  to  attend  at  a  additional  nominal  fee. 

The  topic  the  Executive  Committee  has  selected  for  this  22nd  Annual  National 
T&E  Conference  is  very  timely  and  is  one  that  requires  answers  and  action.  As 
always,  we  have  scheduled  an  outstanding  roster  of  nationally  known  speakers, 
including  those  drawn  from  industry  military  civil  service  and  academia  who 
will  be  addressing  our  theme  issues  from  various  directions. 

The  tutorials  scheduled  for  Monday  are  offered  at  no  additional  charge  to  those 
who  are  registered  for  this  conference.  We  encourage  you  to  avail  yourself  of 
this  additional  opportunity  to  learn  in  more  depth  about  a  number  of  T&E 
initiatives. 

Also,  we  are  again  going  to  recognize  those  who  are  worthy  of  commendation 
for  their  achievements  in  the  world  of  test  and  evaluation.  Plan  to  join  us  for 
both  award  events,  the  first  on  Tuesday  evening  and  the  second  on  Wednesday 
noon. 

We  hope  that  you'll  also  spend  time  in  the  display  area  becoming  aware  of  what 
the  T&E  marketplace  has  to  offer  in  the  way  of  products,  services  and  programs. 

Jacksonville,  FL  is  a  beautiful  city.  Stop  by  the  NDIA  registration  desk  for 
information  on  what  Jacksonville  has  to  offer. 

Jim  O'Bryon,  Chairman,  NDIA  T&E  Division 


NDIA  TEST  AND  EVALUATION  DIVISION 


The  Test  and  Evaluation  Division  was  initially  established  under  the  auspices  of  the  American  Defense  Preparedness  Association 
(ADPA).  With  the  merger  of  National  Security  Industrial  Association  and  the  American  Defense  Preparedness  Association  on 
March  1,  1997,  the  T&E  Division  and  its  responsibilities  were  transferred  to  the  National  Defense  Industrial  Association.  It  is  one 
of  30  divisions  of  the  NDIA. 


NDIA  TEST  AND  EVALUATION  DIVISION  OBJECTIVES 


NDIA’s  principal  missions  are  to  improve  weapons  technology,  improve  defense  management,  and  maintain  a  strong 
science-industry-defense  team  continually  responsive  to  all  needs  of  the  research,  development,  test  and  evaluation,  production, 
logistics  and  management  phase  of  national  preparedness.  It  provides  a  forum  for  the  exchange  of  ideas  and  information 
between  its  members  and  government  agencies  through  a  network  of  divisions,  chapters,  national  and  local  meetings,  and 
conferences  and  visits  to  Department  of  Defense  installations.  Consistent  with,  and  in  furtherance  of,  the  By-Laws  and  policies  of 
the  NDIA,  the  objectives  of  the  T&E  Division  with  respect  to  those  government  elements  appropriate  to  its  field  of  interest  shall 
be: 


Ensure  the  continuation,  responsiveness,  and  where  necessary,  the  revitalization  of  the  industrial  base  to  support  our 

national  security  by: 

=  Maintaining  effective  liaison  with  the  Department  of  Defense,  Executive  and  Legislative  Branches  of  the  Government, 
and  other  governmental  departments  and  agencies  to  apprise  appropriate  representatives  of  these  organizations  of 
T&E  Division  activities  and  keep  abreast  of  current  and  future  developments. 

=  Reviewing,  evaluating,  and  providing  recommendations  with  regard  to  government  policies,  practices,  directives,  and 
specifications  including  pending  legislation  concerning  life  cycle  logistic  support,  supply  chain  integration,  strategic  and 
tactical  mobility,  and  business  processes. 

=  Serving  as  a  catalyst  for  community  dialogue  among  T&E  communities  within  government,  industry  and  academia  as 
well  as  provide  an  overarching  means  for  dynamic  interchanges  of  information  with  other  functional  areas. 

=  Sponsoring  government/industry  meetings,  seminars,  and  symposia  to  provide  effective  communications  between 

government  and  industry  relative  to  management  activities  for  which  the  T&E  Division  is  responsible  under  this  matter. 

=  Maintaining  viable  liaison  with  other  divisions  of  the  NDIA,  other  associations,  and  national  -  level  industry  coalitions 
for  the  purpose  of  exchanging  information  of  mutual  interest  and  coordinating  and  cooperating  on  activities  requiring 
Joint  participation. 


Developing  strategic  directions  effecting  T&E  partnerships  between  national  security  industries  and  government  and  to 
formulate  proposals  for  national  policies,  addressing  their  impact  on  the  national  security  industrial  base. 

Establishing  and  maintaining  a  productive  proactive  dialogue  with  the  national  news  media. 

Fostering  learning  of  the  basics  and  advanced  concepts  and  techniques  of  T&E  within  the  defense  industry  including  the 
cross  -  pollination  of  commercial  approaches  to  T&E  for  the  defense  community. 

Commemorating  and  recognizing  services  rendered  by  individuals,  companies,  and  government  agencies  in  meritorious 
acts  related  to  activities  enumerated  above. 


The  Walter  W.  Hollis  Award 


The  Walter  W.  Hollis  Award  is  presented  annually  in  recognition  of  lifetime  contributions  and  achievement  in  the  area  of  defense 
Test  and  Evaluation.  The  Award  is  presented  in  the  name  of  Walter  W.  Hollis  who  is  recognized  for  his  dedicated  and 
long-standing  service  in  the  field  of  Test  &  Evaluation. 


Previous  Recipients 

2005  Hon  Thomas  Christie,  DOT&E,  OSD 
2004  Dr.  Marion  L.  Williams,  Headquarters  AFOTEC 
2003  Mr.  James  W.  Fasig,  Aberdeen  Test  Center 


2002  Mr.  G.  Thomas  Castino,  Underwriters  Laboratories,  Inc. 

2001  Hon  Philip  E.  Coyle  III,  DOT&E,  OSD 

2000  Mr.  Walter  W.  Hollis,  Department  of  the  Army 


MONDAY 

TUESDAY 

WEDNESDAY 

THURSDAY 

March  6 

March  7 

March  8 

March  9 

Registration  Hours 


7:00  AM  -  5:00  PM 


7:00  AM  -  6:30  PM 


7:00  AM -5:15  PM 


7:00  AM -3:00  PM 


Display  Hours 


Set-Up 

11:00  AM -5:00  PM 


7:00 

AM- 

8:00  AM 

10:15 

AM- 

10:45  AM 

12:15 

PM 

-1:15  PM 

2:45 

PM- 

3:15  PM 

6:30 

PM- 

8:00  PM 

7:00  AM  -  8:00  AM 


9:45  AM -10:15  AM 


11:45  AM- 1:15  PM 


2:45  PM -3:15  PM 


7:00  AM -8:00  AM 


10:00  AM -10:45  AM 


12:00  PM -1:00  PM 


2:30  PM -3:00  PM 


“The  Department  of  Defense  finds  this  event  meets  the  minimum  regulatory  standards  for  attendance  by  DoD  employees. 

This  finding  does  not  constitute  a  blanket  approval  or  endorsement  for  attendance.  Individual  DoD  component  commands  or 
organizations  are  responsible  for  approving  attendance  of  its  DoD  employees  based  on  mission  requirements  and  DoD  regulations. 


AGENDA 


MONDAY,  MARCH  6,  2006 


7:00  AM  -  5:00  PM 
11:00  AM -5:00  PM 
1:00  PM -4:45  PM 


Conference  Registration 
Displays  Set-Up 

Tutorials  (These  tutorials  are  offered  at  no  extra  cost  to  all  registrants) 


TRACK  1 

1:00  PM -2:45  PM 

Break 

3:00  PM -4:45  PM 

“How  to  Improve  the  Effectiveness,  Efficiency,  & 
Integration  of  Test  &  Evaluation  (T&E)  &  Modeling  & 
Simulation  (M&S),”  Dr.  Mark  Kiemele,  Air  Academy 
Associates  (PART  1) 

“How  to  Improve  the  Effectiveness,  Efficiency,  & 
Integration  of  Test  &  Evaluation  (T&E)  &  Modeling  & 
Simulation  (M&S),”  Dr.  Mark  Kiemele,  Air  Academy 
Associates  (PART  2) 

TRACK  II 

“Putting  the  Missions  and  Means  Framework  (MM F) 
to  Work  to  Drive  Analysis  -  Getting  to  the  “So  What” 
by  Asking  the  Right  Questions  about  the  Mission  and 
Generating  Complete  and  Relevant  Answers.  A  Practical 
Tutorial  on  MMF  Application,”  LTC  Britt  Bray,  USA  (Ret), 
Dynamics  Research  Corporation,  Inc.;  Mr.  Bill  Yeakel, 
President,  ORSA  Corporation  (PART  1) 

“Putting  the  Missions  and  Means  Framework  (MMF) 
to  Work  to  Drive  Analysis  -  Getting  to  the  “So  What” 
by  Asking  the  Right  Questions  about  the  Mission  and 
Generating  Complete  and  Relevant  Answers.  A  Practical 
Tutorial  on  MMF  Application,”  LTC  Britt  Bray,  USA  (Ret), 
Dynamics  Research  Corporation,  Inc.;  Mr.  Bill  Yeakel, 
President,  ORSA  Corporation  (PART  2) 

4:45  PM  Tutorial  Sessions  Close 

5:00  PM  Conference  Registration  Closes  (Registration  will  re-open  tomorrow  at  7:00  AM.) 

There  is  no  scheduled  function  this  evening.  Please  stop  by  the  registration  desk  for  a  listing  of  local  dining  options, 
entertainment,  shopping,  etc. 


I  I  Your  attention  please:  Survey  sheets  will  be  given  to  every  conference  registrant  upon  arrival.  The  survey  sheets  are 
intended  to  gather  your  thoughts  and  ideas  regarding  this  NDIA  T&E  Conference.  Registrants  are  requested  to  thoughtfully 
complete  these  surveys  and  return  them  to  the  NDIA  desk  prior  to  their  departure.  The  results  of  these  surveys  will  be  used  to 
serve  as  a  basis  in  planning  for  future  conferences  and  should  be  completed. 


Jacksonville  Skyline 


TUESDAY,  MARCH  7,  2006 


7:00  AM 
7:00  AM 
8:00  AM 
8:05  AM 
8:10  AM 
8:15  AM 
8:55  AM 

9:30  AM 

10:15  AM 
10:45  AM 


12:15  PM 


1:15  PM 


Registration  Opens 
Continental  Breakfast  in  Display  Area 
Call  to  Order,  Mr.  Sam  Campagna,  NDIA 
Presentation  of  the  Colors  and  National  Anthem 


Welcome  and  Opening  Comments,  Mr.  James  O'Bryon,  Chairman  NDIA  Test  and  Evaluation  Division 

Conference  Keynote  Address,  Gen  Larry  Welch,  USAF  (Ret),  former  President,  IDA  &  former  Chief  of  Staff,  USAF 

“M&S  and  Testing:  Collision  Course  or  Happy  Marriage?, "  Hon  Philip  E.  Coyle,  III,  former  Director, 
Operational  Test  &  Evaluation  (DOT&E),  OSD 

Session  #1 

PRESENTATION  OF  THE  ISSUES 

Host:  Mr.  Larry  Graviss,  Chairman,  NDIA  ICOTE  Committee,  Jacobs  Sverdrup 

ICOTE  Editorial  Report:  “Assessment  of  Large-Scale  Distributed  Engineering  M&S  Plants,"  Mr.  Jim 
Sandberg,  Northrop  Grumman;  Mr.  Michael  Crisp,  ODOT&E 

Break  in  Display  Area 


Panel 

“Addressing  Non  Technical  Obstacles  to  Effective  M&S  Connectivity  in  Support  of  T&E” 

Panel  Chair:  Mr.  John  lllgen,  Northop  Grumman 

Panelists: 

Mr.  Rick  Cozby,  DTC 
Mr.  Jack  Sheehan,  FCSCTO 
Mr.  Augustine  Ponturiero 
Mr.  Peter  Shaw 
Dr.  Paul  H.  Deitz,  ARL 

Luncheon 


Session  #2 

Host:  Mr.  Dick  Dickson,  Tybrin  Corporation 


Panel 

“DT&E’s  Perspectives  on  Making  M&S  and  T&E  Better  Partners” 

Panel  Chair:  Col  Richard  Stuckey,  USAF,  Acting  Deputy  Director,  DT&E,  0USD(AT&L) 
Panelists  (Government  Representatives): 

Dr.  George  Kai’liwai,  Technical  Advisor,  AFFTC  Edwards  AFB,  CA 
Mr.  Thomas  Berard,  Director,  WSMR 
Panelists  (Industry  Representatives): 

Mr.  Steven  Cameron,  Lockheed  Martin 
Mr.  James  A.  Flazlett,  Raytheon 


2:45  PM 


Break  in  Display  Area 


TUESDAY,  MARCH  7,  2006  (Continued) 


Session  #3 

Host:  Dr.  Juan  Vitali,  ATEC 

3:15  PM  “DOT&E  Perspectives,"  Mr.  David  Duma,  Acting  Director,  Director  Operational  Test  &  Evaluation 

3:45  PM  "An  LFT&E  Perspective  on  Making  M&S  and  T&E  Better  Partners,”  Dr.  Lowell  Tonnessen,  IDA 

4:15  PM  “Modeling  the  FCS  Brigade  Combat  Team  in  a  Stability  and  Reconstruction  Operations  Mission 

Implications  for  T&E  Planning,"  LTC  Britt  Bray,  USA  (Ret),  Dynamics  Research  Corporation 

4:45  PM  “Transformation  In  TD&E,"  Mr.  Bill  Yeakel,  President,  ORSA  Corporation 

5:15  PM  Afternoon  Session  Closes 


6:00  PM  Reception 

6:30  PM  Flonors  Banquet-  The  cost  of  the  banquet  is  included  in  the  conference  registron  fee.  Guests  (non¬ 

conference  registrants)  are  also  welcome  to  attend  this  banquet  for  an  additional  fee.  Reservations  must  be 

made  at  the  NDIA  registration  desk  by  Monday,  March  6,  2006  at  5:00  PM. 


Guest  of  Honor 

This  event  has  historically  been  a  highlight  of  our  National  NDIA  T&E  Division  Annual  Conference  and  this  year  is  no  exception.  Our 
2006  recipient  of  the  Walter  Hollis  Award  for  Lifetime  Achievement  in  Test  and  Evaluation  is  RADM  Charles  “Bert”  Johnston,  USN,  (Ret). 

RADM  Bert  Johnston’s  T&E  involvement  extends  more  than  30  years  including  time  with  the  US  Navy  as  a  Test  Pilot, 
Program  Manager,  Systems  Engineer,  and  a  host  of  other  positions  in  the  U.S.  and  abroad.  He  has  served  with 
distinction  in  senior  testing  positions  in  Australia,  Pax  River,  China  Lake  and  more  recently  as  Commander,  Naval 
Weapons  Center,  Weapons  Division,  China  Lake,  CA. 

In  January  2002  he  became  Vice  Commander  for  the  Naval  Air  Systems  Command.  He  continued  to  stay  involved 
in  testing,  supporting  the  integration  of  joint  capabilities  and  the  expanded  use  of  test  facilities  for  rehearsal, 
demonstration,  training,  and  experimentation.  His  transition  from  the  Navy  in  the  Spring  of  2005  wasn’t  the  end 
of  his  involvement  with  testing.  He  now  serves  as  Vice  President  and  General  Manager  for  the  Systems  Engineering  Division  of  Wyle 
Laboratories,  Naval  Aviation’s  largest  support  contractor.  Among  his  responsibilities  are  Wyle’s  Flight  Test  Operation  and  the  Telemetry 
Data  Systems.  Congratulations  from  all  of  us  for  a  job  done  well!! 


VADM  Bennitt  (Ret),  USN  has  flown  over  4000  hours  in  more  than  fifty  different  aircraft  types  and  models.  He 
graduated  from  the  U.S  Naval  Academy  in  1964  and  was  designated  a  Naval  Aviator  eighteen  months  later.  After 
two  combat  cruises  and  222  combat  missions  with  Attack  Squadron  94,  flying  the  A-4  Skyhawk  aboard  USS 
HANCOCK  and  USS  BONNE  HOMME  RICHARD,  he  attended  the  U.S.  Naval  Test  Pilot  School  and  served  as  an 
Aircraft  Carrier  Suitability  Test  Pilot  at  the  Naval  Air  Test  Center.  This  assignment  was  followed  by  a  tour  as  a 
flight  test  tutor  at  the  Empire  Test  Pilot  School  at  Boscombe  Down,  England,  during  which  he  was  commended  by 
the  Queen  of  England  for  valuable  service  in  the  air  as  the  school's  Stall/Spin  Program  Manager  and  as  one  of  the 
development  test  pilots  on  England’s  first  variable  stability  training  aircraft. 

Vice  Admiral  Bennitt  was  later  assigned  to  Washington,  DC  where  he  served  as  Acting  Director  of  Aircraft  Carrier  and  Air  Station 
Programs  in  the  office  of  the  Chief  of  Naval  Operations.  He  then  received  orders  as  Commanding  Officer,  USS  NIMITZ,  taking 
command  on  August  28,  1987.  In  September  1989,  Vice  Admiral  Bennitt  was  assigned  as  Deputy  Director,  Test  and  Evaluation  and 
Technology  Requirements  (OP-091B)  in  the  office  of  the  Chief  of  Naval  Operations.  In  November  1991,  he  returned  to  the  fleet  to 
take  command  of  Carrier  Group  FIVE  and  Battle  Force  Seventh  Fleet,  home-ported  in  Yokosuka,  Japan.  In  October  1993,  Vice  Admiral 
Bennitt  was  assigned  as  Director,  Air  Warfare  Division  in  the  office  of  the  Chief  of  Naval  Operations.  He  then  assumed  command  as 
Commander,  Naval  Air  Force,  U.S.  Pacific  Fleet  in  January  1996.  For  his  contributions  to  flight  safety  during  that  tour  the  International 
Flight  Safety  Foundation  selected  him  as  the  1998  recipient  of  the  Laura  Taber  Barbour  International  Air  Safety  Award. 

Mr.  Bennitt  is  now  Executive  Vice  President  and  General  Manager  of  the  Wyle  Aerospace  Group.  He  also  serves  as  a  National  Director 
of  the  Navy  League  of  the  United  States. 

Plan  to  attend  our  banquet  and  enjoy  the  evening  and  pay  tribute  to  this  year’s  honoree. 


Guest  Speaker 


WEDNESDAY,  MARCH  8,  2006 


Registration  Opens 
Continental  Breakfast  in  Display  Area 

Call  to  Order  and  Administrative  Announcements,  Mr.  Sam  Campagna,  NDIA 
Focus  on  our  Customers  (Video) 

Session  #4 

LOOKING  AT  THE  FACTS 
Host:  Mr.  Bill  Yeakel,  ORSA  Corporation 

DMSO  PANEL 

“The  Challenge  of  Using  M&S  in  T&E” 

Panel  Chair:  COL  Jerry  Glasow,  USA,  Director,  DMSO 
Panelists: 

Mr.  Michael  Crisp,  ODOT&E 
Mr.  Chris  Roust,  TRMC,  OSD 
TBD,  ATEC 

Mr.  Jerry  Kitchen,  AFOTEC 
Mr.  Steven  Whitehead,  OPTEVFOR 

9:45  AM  W3  Break  in  Display  Area 

Qpccinn  ff  R 

MILITARY  SERVICE  PERSPECTIVES 
Host:  Dr.  Ernest  Seglie,  DOT&E 

10:15  AM  “Perspective  from  the  Army,”  Mr.  Walter  Hollis,  DUSA/OR 

1 0:45  AM  “Perspective  from  the  Navy,”  RADM  Bill  McCarthy,  USN,  COMOPTEVFOR 

1 1 : 1 5  AM  “Perspective  from  the  Air  Force,”  Mr.  Jack  Manclark,  AF/Test  &  Evaluation  Directorate 


7:00  AM 
7:00  AM 
8:00  AM 
8:00  AM 

8:15  AM 


11:45  AM 

Annual  Awards  Luncheon 

The  Tester  of  the  Year  Awards  are  presented  annually  to  civilian  and  military  personnel  supporting 
all  of  the  military  services  and  OSD.  For  each  military  service  and  OSD,  an  award  is  presented  to  the 
outstanding  defense  civilian,  outstanding  military,  and  outstanding  supporting  contractor. 

Award  Winners 

Military 

Civilian 

Contractors 

MAJ  Sandy  Vann,  USA 

Mr.  Mark  Simon,  Army  Civilian 

Mr.  Freddie  Lee,  Army  Contractor 

MAJ  Victoria  Miralda,  USA, 

OSD 

Mr.  Tracy  Sheppard,  OSD  Civilian 

Mr.  Steven  Seiden,  OSD  Contractor 

Maj  James  Colebank,  USAF 

Mr.  Kenneth  Lawrence,  USAF  Civilian 

Ms.  Katherine  Hallett,  USAF  Contractor 

CDR  Chris  Holmes,  USN 

Mr.  Dean  Kimelheim,  Navy  Civilian 

Mr.  Rollin  Sublett,  Navy  Contractor 

LtCol  Christopher  Seymour,  USMC 

Mr.  Bruce  Leaman,  Marine  Corps  Civilian 

Mr.  Gary  Evans,  Marine  Corps  Contractor 

1:15  PM 
1:45  PM 
2:15  PM 

2:45  PM 

3:15  PM 
3:45  PM 
4:15  PM 
4:45  PM 
5:15  PM 


Session  #6 

M&S  STUDIES  IN  THE  CONTEXT  OF  T&E  AND  ACQUISITION 
Host:  Dr.  Lowell  Tonnessen,  IDA 

“Diagnosis  of  the  Problem:  What  Prior  Studies  Have  to  Say,"  Dr.  James  E.  Coolahan,  APL,  JHU 
“The  Joint  M&S  Study”,  Dr.  Frank  Gray,  AFOTEC 

“Open  Air  Testing  vs.  Modeling  and  Simulation:  At  Last  Peaceful  Coexistence?,"  RADM  Charles  “Bert” 
Johnston,  USN  (Ret),  Vice  President  &  General  Manager,  Wyle  Laboratories 

Break  in  Display  Area 


Session  #7 

Host:  Mr.  John  lllgen,  Northrop  Grumman 

“INDIA  Industry  Perspectives,”  Lt  Gen  Larry  Farrell  Jr.,  USAF  (Ret),  President  &  CEO,  NDIA 

“Open  Technology  Development  &  Testing,"  Mr.  John  Scott,  Selection  Pressure,  LLC 

“Modeling  and  Simulation  of  Systems,”  Mr.  Vincent  M.  Ortiz,  AV\N  Technologies 

“M&S  /  T&E  Event  -  Working  Together:  AUV  Fest  2005  ISR  Scenario,"  Mr.  Reid  Johnson,  NUWC,  Keyport 

Afternoon  Session  Closes 


Paddle  Boat  Reception 

You  are  invited  to  join  us  for  a  reception  wtih  cocktails,  heavy  hors  d’oeuvres,  and  a  carving  station 
aboard  the  Lady  St.  Johns  hosted  by  NDIA  and  AVW  Technologies. 

Only  $25  per  person.  Guests  are  welcome  to  attend. 

Boarding  time  6:00  PM  -  6:30  PM  at  the  Hyatt  Regency  Jacksonville  dock  located  behind  the  hotel. 


Innovative  Solutions  to  the  Challenges  of  the  Future 

AVW  Technologies,  Inc.  is  a  small,  veteran-owned  business  that  provides  Professional  Engineering  Services,  Test  and  Evaluation  Support,  and 
Shipbuilding  Engineering  Management  Consulting.  Our  efforts  to  date  have  primarily  been  focused  on  surface  ship  acquisition,  from  Design 
through  Production  and  Test  and  Lifecycle  Management.  AVW  is  also  a  recognized  leader  in  the  use  of  Integrated  Product  Data  Environments 
and  Digital  Product  Models  to  support  Ship  Acquisition,  as  well  as  the  use  of  Modeling  and  Simulation  technologies  to  support  Design, 
Production,  and  Test  and  Evaluation. 

At  AVW  Technologies,  our  team  of  former  Naval  Professionals  is  skilled  in  Acquisition,  Engineering  and  T&E  Management,  and  Naval  Warfare. 

Our  staff  includes  many  former  Operational  Test  Directors,  Operational  Test  Coordinators,  and  military  analysts  with  matrixed  military  experience 
across  technical/warfare  disciplines,  including: 

•  Surface  combatants,  amphibs,  CVN,  aviation,  E-2,  TACAIR,  USMC,  staffs 

•  Combat  systems,  HM&E,  suitability,  AAW,  ASW,  MIW,  STW/LAW,  SUW,  C4ISR,  IW,  AMW/EXW  +  Seabasing,  AT-FP,  coastal  warfare 

•  CT/DT/LFT/OT,  Systems  Engineering,  JCIDS/Requirements,  IPT  Management 

•  M&S  Management 

Our  company  takes  pride  in  hiring  only  the  most  qualified  personnel  and  making  them  available  to  our  customers  at  reasonable  rates.  We 
endeavor  to  remain  current  with  the  Doctrine  and  Policies  of  today’s  revolution  in  military  affairs.  We  are  committed  to  supporting  our  clients  to 
our  fullest  capabilities,  and  we  encourage  all  our  staff  to  go  the  extra  mile  to  meet  our  customers’  needs. 


AVW 

TKCH  M  OT ,00  I  KS 


6:00  PM  -  9:00  PM 


$1.  ofejnis  •  Accommodates  300 
tH&maUU  2m  •  Accommodates  ISO  | 


AVW  Technologies,  Inc. 


We  are  pleased  to  have  Vincent  Ortiz  of  AVW  Technologies  presenting  several  case  studies  based  on  our  work  on  the  LPD  17  PRA  Testbed 
during  the  “M&S  of  Systems  of  Systems”,  “W&  A  -  Does  it  Work?  Can  We  Make  it  Work?”,  and  “How  Can  M&S  Optimize  Its  Use  of  T&E  Data  and 
Vice  Versa”  sessions.  Please  stop  by  our  exhibition  booth  to  learn  more. 


THURSDAY,  MARCH  9,  2006 


7:00  AM 

Registration  Opens 

7:00  AM 

Continental  Breakfast  in  Display  Area 

8:00  AM 

Call  to  Order  and  Administrative  Announcements,  Mr.  Sam  Campagna,  NDIA 

Session  #8 

Host:  Dr.  Paul  H.  Deitz,  ARL 

8:05  AM 

“Using  the  Missions  and  Means  Framework  to  Generate  Cost-  Effective  Live-Fire  Test  and  Evaluation 
Strategies" 

Presenters: 

Dr.  Martha  K.  Nelson,  Franklin  &  Marshall  College  &  Dennis  C.  Bely,  US  Army  Research 
Laboratory 

9:00  AM 

( 

“Modeling  and  Simulation  to  Advance  Test  and  Evaluation,"  Dr.  Sheldon  Tieszen  &  Dr.  Lou  Gritzo, 
Sandia  National  Laboratories 

10:00  AM 

w> 

Break  in  Display  Area 

10:45  AM 

Begin  Parallel  Tracks 

TRACK  1 

T&E  /  M&S  Architecture 

10:45  AM 

11:15  AM 

“Test  and  Training  Architecture,  TENA, 

Offers  Range  Interoperability  and  Resource 
Reuse  Solutions  for  DoD  Test  and  Training 
Ranges,”  Mr.  Gene  Hudgins,  BAE  Systems 

“LPD  17  PRA  Testbed:  A  Case  Study  of 

T&E  Data  Supporting  Architecture,”  Mr. 
Vincent  M.  Ortiz,  AVW  Technologies 

TRACK  II 

(WA)  “V,V&A”  Issues  to  Make  T&E/M&S  Work  Effectively 

10:45  AM 

11:15  AM 

“LPD  PRA  Testbed  W&A  Database: 

A  disciplined  Approach  for  W&A,"  Mr. 
Vincent  M.  Ortiz,  AVW  Technologies 

“Modeling  and  Testing  of  Civilian  GPS 
Systems  and  Potential  Vulnerabilities,"  2Lt 
Shawn  Littleton,  USAF,  Holloman  AFB 

12:00  PM  Lunch 


THURSDAY,  MARCH  9,  2006  (Continued) 


1 :00  PM  Parallel  Tracks  Continued 


TRACK  1 

Policy  Proposals 

1:00  PM 

1:30  PM 

2:00  PM 

“27  Steps  to  Success:  The  DoD 
Acquisition  M&S  Master  Plan,”  Mr. 
Fred  Myers,  OUSD(AT&L)/DS/SE 

“No  More  Second  Opinions:  Oganizing 

DoD  M&S  within  the  Acquisition  Cycle, 

Mr.  James  O’Bryon,  The  O’Bryon  Group 

“Optimizing  Cost  and  Time 
Effectiveness  of  Test  and 

Evaluation  Using  Knowledge-Based 
Simulations,”  Dr.  Keith  Bradley,  LLNL 

TRACK  II 

Examples  of  Attempts  to  Get  Mutual  Support  between  T&E  and  M&S 

1:00  PM 

1:20  PM 

1:40  PM 

2:00  PM 

“Intelligent  Munitions 
System:  Balancing  T&E 
and  M&S,”  Mr.  William  J. 
Bush,  ARDEC,  Picatinny 
Arsenal 

"Integration  of  Transport 
Modeling  with  Test  and 
Valuation  in  CBR  Building 
Protection  Systems,”  Mr. 

James  Risser,  Battelle 

“InterTEC:  Testing  Joint  Mission 
Threats  in  a  TENA  Based 
Environment,”  Mr.  Robert 
Heilman,  Pt.  Mugu,  CA 

“Dual  Thrust  Modified 
Smokey  SAMS  For 

Low  Cost  Testing  and 
Simulation  of  Aircraft 
Missile  Warning  and 

Laser  IRCM  Systems,” 

Mr.  William  R.  Taylor, 

( WPAFB ) 

2:30  PM 


3:00  PM 


Break  in  Display  Area 


Session  #9 

T&E  /  M&S  Infrastructure  Issues 
Mr.  Jim  O’Bryon,  NDIA  T&E  Division  Chair,  The  O’Bryon  Group 

General  Session 


“CAW  (Center  for  Asymmetric  Warfare)  Use  of  M&S  in  Maritime  HLS/HLD/ATFP  Test  and  Training 
Exercises,”  Mr.  Arman  Tchoubineh,  NAWC  WD,  Pt.  Mugu,  CA 

“Knowledge  is  Power  -  Joint  Datalink  Information  Combat  Execution  (JDICE)  Joint  Test  and  Evaluation,” 
Mr.  Clarence  Stone,  JDICE  JT&E  OSD 

“Building  An  Extensable  Virtual  Sensor  Network,”  Mr.  Joshua  J.  Pressnell,  RTI 

“Commercial  Air  Services;  Live  Test  Assets  for  Electronic  Warfare,”  Mr.  Stan  Ulkoski,  PMA-207D2 
Aircraft  Operation  Manager  J.F.Taylor,  Inc.,  NAVAIR  support  contractor;  Ms.  Linda  Barrett  (JFTI) 

5:00  PM  Conference  Synopsis  and  Closing  Remarks,  Mr.  James  O'Bryon 

(Reminder:  Please  submit  your  completed  Conference  Survey  Sheets) 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


2111  Wilson  Blvd.f  Suite  400 
Arlington,  VA  22201 
www.ndia.org 


Thank  you  to  our 
Promotional  Partners 

AVW 
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22nd  Annual  National  Test  &  Evaluation  Conference 
“Improving  the  Test  and  Evaluation  of  Defense  and  Homeland  Security  Systems  Through 
Effective  Development  and  Intelligent  Application  of  Modeling  and  Simulation.” 

March  6-9,  2006 


22nd  Annual  National  Test  &  Evaluation  Conference,  Jacksonville,  Florida 


Optimizing  Cost  &  Time  Effectiveness  of 
Test  &  Evaluation  Using  Knowledge- 
Based  Simulations 


Click  to  f  J!x  "--x - 

Dr.  Keith  S.  Bradley 

Lawrence  Livermore  National  Laboratory 


March  9,  2006 


This  work  was  performed  under  the  auspices  of  the  U.S.  Department  of  Energy  by  the  University  of  California 
Lawrence  Livermore  National  Laboratory  under  Contract  No.  W-7405-Eng-48. 


Modeling  &  Simulation  will  never  replace  Test  & 
Evaluation 


•  There  is  no  substitute  for  experience 

•  Even  the  most  accurate  simulations  require  understanding  that  is 
either  obtained  or  validated  by  experiment 

•  Simulations  that  are  not  validated  (W&A)  are  about  as  useful  as 
testing  and  evaluation  techniques  that  are  not  validated 


Intelligently  applied,  knowledge-based 
simulations  can  enhance  the  development 
and  acquisition  process 


060309.T&E. Jacksonville. KSB  2 


In  1992,  the  Department  of  Energy  faced  a 
perplexing  new  challenge 


•  PRESIDENTIAL  DIRECTIVE:  Certify  the  performance,  safety  and 
reliability  of  the  enduring  nuclear  deterrent  WITHOUT  TESTING. 


060309.T&E. Jacksonville. KSB  3 


What  are  knowledge-based  simulations? 


Empirical 

Knowledqe-Based 

Simulations 

Simulations 

•  Describe  outcomes 

•  Predict  outcomes 

-  Use  fits  or  tables  derived 

-  Use  equations  derived  from 

from  data 

understanding  of  governing 
mechanisms 

-  For  quick  interpolations 

between  data  points  when 

-  For  reasonable  excursions 

variances  are  intuitive 

beyond  and  between  data 
points  when  variances  aren’t 
intuitive 

060309.T&E. Jacksonville. KSB  4 


Knowledge-based  simulations  can  enhance  test  & 
evaluation  in  different  ways 


Design  Tests 

Interpret  Tests 

Supplement  Tests 

Where  to  “look” 

What  to  “look  for” 
What  to  vary 
Reduce  risk  for  failure 


Unravel  confusing  data  sets  and 
provide  insight  about  the 
important  phenomena  that 
produce  them 


Provide  insight  where 
experiments  are  prohibitively 
expensive  or  physically 
impossible 


060309.T&E. Jacksonville. KSB  5 


The  missile  defense  community  uses  knowledge-based 
simulations  to  assess  kinetic  intercept  lethality 


Standard  Missile  -3  (SM-3,  AEGIS) 


Simulated  Ground-Based  Midcourse 
Defense  (GMD)  Intercept 


060309.T&E. Jacksonville. KSB  6 


Debris  patterns  can  be  correlated  with  real-time  hit 
assessment  immediately  following  a  flight  test 


Mote  nob 


2  m 

3  ■ 

4  ■ 

3  ■ 


Global  Maximum;  3.1De-D1,  Nodal  13G76 
Global  Minimum;  O.OGe+GG,  Nodal  64213 
Displacement  Scale;  1-0/ 1.0/ 1-0 


Velocity  Magnitude 
2*50e-01  - 
2 G0e-01  - 
1*50e-01  - 


fr  1 0hs  2  km/s 
t  =  0.00000e  +  00 


Y_J 


f r  1 0il*  6,8  km/s 
t  -  0-00000e+00 
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Full-Scale  Sled  Track  Test 


Missile  intercept  simulations 
contribute  jn  all  three  categories 


Scaled  Light  Gas  Gun  Test 


060309.T&E. Jacksonville. KSB  8 


Penetration  continues  to  be  an  important 
mechanism  in  weapon  jethality 


Penetration  of  Armor 


► 


before 


Defeat  of  Hard  &  Deeply 
Buried  Targets 
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Predicting  lethality  requires  predicting  debris 


before 


(note:  debris  hidden 
In  this  snapshot) 


willy 

Thu  Aug  18  )4l46:2 


This  debris  is 
usually  what  kills 
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Generating  and  propagating  fragments  is  necessary 
to  assess  collateral  effects 


High  explosive 
detonating  " 
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Ground  shock  (lethality)  is  significantly 
enhanced  in  a  buried  detonation 


-BLU  116  Advanced  Unitary 
Penetrator 
-Length:  2.4  m 
-Diameter:  25.4  cm 
-Thickness:  50  mm 
-Weight:  770  kg 
-HE  weight:  65  kg 


•  Consider  the  scenario  of 
sequentially  delivered  penetrators 

•  Penetration  is  less  effective  with 
successive  attempts 

•  Can  this  process  be  optimized? 


0  50  100  150 


Time  (milliseconds) 
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Insight  gained  from  knowledge-based 
simulations  suggests  amodrfied  approach 


“Clearing” 
bomb  debris 
from  the  crater 
led  to  a  factor  of 
1.5  increase  in 
penetration 


Penetration  at  the 
surface  is  4  times 
deeper  than  penetration 
inside  a  crater. 


Penetration  depth  in  shock-conditioned  granite 
was  3  times  greater  than  it  was  in  virgin  rock 


Alternating  penetrators  with  conventional  bombs 

may  be  more  effective 


060309.T&E.  Jacksonville.  KSB  13 


Knowledge-based  simulations  provide  true  value 
for  defense  acquisition 


•  Though  M&S  cannot  replace  T&E,  they  provide  enhancements  by 


•  Optimizing  the  design  of  developmental  and  operational  tests 

•  Provide  guidance  in  interpreting  the  results 

•  Augment  data  sets  when  a  test-only  approach  is  cost-prohibitive, 
time-prohibitive  or  physically  impossible 


•  These  enhancements  permit  an  additional  means  for  exploring 
alternatives,  modifications  and  “what  if  s” 


•  Intelligently  applied,  knowledge-based  simulations  can  accelerate 
discoveries  and  minimize  risks  in  the  development  process 


060309.T&E.  Jacksonville.  KSB  14 


The  22nd  Annual  National 
Test  &  Evaluation  Forum 


Sponsored  by  the  National  Defense  Industrial 
Association  Test  &  Evaluation  Division 

March  7,  2006 

\ 

Modeling  the  FCS  Brigade  Combat  Team  in  a 
Stability  and  Reconstruction  Operations  Mission 
-  Implications  for  T&E  Planning 


Held  at  The  Hyatt  Regency  Jacksonville, 
Jacksonville,  FL,  6-9  March,  2006. 


LTC  (R)  Britt  E.  Bray 
Dynamics  Research  Corp. 
bbray@drc.com 


Agenda:  Tuesday,  7  March,  2006 

Modeling  the  FCS  Brigade  Combat  Team  in  a 
Stability  and  Reconstruction  Operations  Mission 

Implications  for  T&E  Planning 


•  Premise 

•  Background 

•  Applying  the  Framework  to  Generate  Questions 

•  Notional  Scenario  Description  and  Mission  Statement 

•  Front  End  Planning  Products 

•  Implications  for  T&E  Planning 

•  S&RO  Challenges  for  T&E 

•  Summary  and  Conclusion 


Premise 


Recent  experience  shows  dramatic  need  for  Army 
organizations  to  provide  capability  to  succeed  in  Phase 
IV,  Stability  and  Reconstruction  Operations. 

Planning  for  T&E  of  FCS  BCT  should  ensure 
Effectiveness,  Suitability  and  Survivability  of  FCS  BCT 
and  its  components  conducting  S&RO  missions  under 
realistic  and  relevant  scenario  driven  conditions. 

Application  of  MMF  principles  to  front  end  T&E 
planning  can  help  to  ensure  that  the  right  questions  are 
asked  and  answered  in  the  design  and  execution  of  the 
T&E  plan. 


Agenda:  Tuesday,  7  March,  2006 

Modeling  the  FCS  Brigade  Combat  Team  in  a 
Stability  and  Reconstruction  Operations  Mission 

Implications  for  T&E  Planning 


•  Premise 

•  Background 

•  Applying  the  Framework  to  Generate  Questions 

•  Notional  Scenario  Description  and  Mission  Statement 

•  Front  End  Planning  Products 

•  Implications  for  T&E  Planning 

•  S&RO  Challenges  for  T&E 

•  Summary  and  Conclusion 


S&RO  -  A  Quick  Review 


FM  3.07  provides  Tier  1  doctrine  for  Stability  and 
Support  Operations 

FCS  O&O  identifies  the  following  Stability,  Support 
and  Security  Missions  for  the  F-BCT: 


-  Peace  Operations 

-  Foreign  Internal  Defense 

-  Security 

-  Combating  Terrorism 

-  Arms  Control 

-  Show  of  Force 

-  Area  Security 

-  Local  Security 


30  June  200 
CHANGE  2  t< 
TRADOC  Pamphlet  52f 

The  United  State: 
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S&RO  -  A  quick  review 

Potential  S&RO  Mission  Types  for  F-BCT 

-  Secure  a  Route 

-  Conduct  Airmobile  Raid 

-  Secure  a  Site  (FOB) 

-  Conduct  Cordon  and  Search 

-  Escort  a  Convoy 

-  Execute  a  TCP 

-  Recon  MSR 

-  Border  Surveillance 


FCS  BCT  O&O  Overview 


Qualities/Ov erarching  C apabilities 

-  Quality  of  Firsts 

-  Modes  of  Operation 

-  Air  Ground  Integration 

-  Ground  Manned  and  Unmanned  teaming 

-  Lethal  and  Nonlethal  Effects 

-  Networked  Fires 

-  Maneuver  Support  Imperatives 

-  Maneuver  Sustainment  in  S&RO  environment 


FCS  BCT  Organization 


Increment  1  Objective 


Unit  of  Action  Design 


2950  @14846.19  STONs 


HHC 


142 


FCS  {Manned}: 


102 

ICV 

49 

C2V 

30 

RflSV 

60 

MC5 

24 

HLOS  Mortar 

18 

HLOS  Cannon 

10 

MV-T 

19 

MV-E 

14 

FRMV 

FCS  {Unmanned}: 

27 

ARV-RSTA 

18 

ARVA(L) 

18 

ARV-Assault 

54 

MULE-T 

60 

NLOS  LS 

81 

8UGV 

6 

Multi -Mssi  on  F 

10 

Lightv^ight  Co 

6 

MULE-CM 

6 

MULE-RT 

BICC 


93 


3X597 


410  193  321 


Trucks: 

111  FTTS-C2 
139  FTTS-SPT 
7  FTTS-Kni^rt 
232  FTT3-M3 
4  FTTS-RMV 
6  FTTS-AMB 
2  FTT3-T  (Treatment) 

67  FTTS-MS  Trailers 
UA  Vs: 

55  UAV  CL  I  L/C  Units 

110  UAV  CL  I  Aerial  Vehicles 
36  UAV  CL  II IVC  Uhits 

36  UAV  CL  II  Aerial  Vehicles 
12  UAV  CL  III  L/C  Units 

46  UAV  CL  III  Aerial  Vehicles 

2  UAV  CL  IVa  L/C  Urits 

6  UAV  CL  IV^  Aerial  Vehicles 

3  UAV  CL  IVb  IVC  Uhits 

16  UAV  CL  IUd  Aerial  Vehicles 
Other: 

12  61 -mm  Mortar 
20  ARH 
3  AAFARS 

2  HTAR5 

3  Camels 

6  POL  racks 


MMF  -  A  quick  review 


11  Fundamental  Elements:  7  levels,  4  operators 
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Applying  the  Framework  to  generate 
questions  about  the  Mission  . . . 


Framework 

Level 

Level  Description  in  Terms 
of  Military  Operations 

Questions  related  to  Test  and  Evaluation 

Level  7: 

Mission, 

Purpose 

What  is  the  Mission  Task, 
Purpose,  and  Endstate  for 
Friendly  and  threat  forces? 

What  is  the  Mission  Task,  Purpose,  and  Endstate  for 
friendly  and  threat  forces?  What  operational  scenario  is  to 
be  used  for  the  test? 

Level  6: 

Context, 

Environment 

What  are  the  conditions 
under  which  the  mission  is 
to  be  performed? 

What  are  the  environmental  factors  that  will  impact  on  the 
mission?  How  do  we  replicate  the  appropriate 
environmental  factors? 

Level  5: 

Index, 

Location/Time 

When,  in  terms  of  current 
or  future  time  period  and 
where,  in  terms  of 
geographic  location  does 
the  mission  occur? 

When  and  where  is  the  mission  to  take  place?  What  events 
(Road  to  War)  led  to  this  point?  When  and  where  will  the 
test  activities  take  place? 

And  the  Means  that  lead  to... 


Framework 

Level 

Level  Description  in  Terms  of 
Military  Operations 

Questions  related  to  Test  and  Evaluation 

Level  4: 

Tasks, 

Operations 

What  operations  and  tasks 
must  be  performed  to 
accomplish  the  mission 

What  types  of  operations  and  what  tasks  must  the  organization 
under  test  perform  in  order  to  accomplish  the  mission?  What  is 
purpose  for  each  operation  and  task?  What  are  the  associated 
MOEs  and  MOPs?  What  conditions  are  in  place  that  could 
affect  task  performance? 

Level  3: 

Functions, 

Capabilities 

What  functions  and 
capabilities  are  required  to 
enable  successful 
accomplishment  of  task 
purpose? 

What  functions  and  capabilities  and  capabilities  are  required  to 
enable  successful  performance  of  the  operations  and  tasks  under 
identified  conditions? 

Level  2: 

Components, 

Forces 

What  equipment,  personnel 
and  forces  are  available  to 
provide  required  functions  and 
capabilities 

What  components  and  forces  are  available  to  perform  the 
mission  and  what  functions  and  capabilities  do  they  provide? 

Level  1: 

Interactions, 

Effects 

What  interactions  must  occur 
to  create  the  desired  effect. 

What  interactions/events  must  be  stimulated/simulated? 

T&E  Planning  and  Coordination 

Questions 


•  Does  a  suitable  scenario  exist  or  will  one  have  to  be  generated? 

•  What  are  the  most  important  capabilities  to  test  and  evaluate? 
Under  what  conditions? 

•  What  operations  and  tasks  require  the  identified  capabilities? 

•  What  components  and  forces  of  the  System  under  test  deliver 
and/or  employ  the  capabilities  identified  above? 

•  What  kind  of  LVC  environment  is  required  to  adequately  support 
observation  and  evaluation  of  the  identified  components/forces 
performing  the  identified  tasks? 

•  Do  resources  and  facilities  currently  exist  to  provide  the  required 
LVC  environment  or  will  some  have  to  be  developed? 
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Notional  Scenario  Situation 


•  FCS  BCT  Operating  as  part  of  Ground 
Component  of  a  Joint  Force  conducting  offensive 
operations  in  Orangeland. 

•  FCS  BCT  Operating  as  part  of  Ground 
Component  of  a  Joint  Force  conducting  offensive 
operations  in  Orangeland. 


Notional  Scenario  Mission  Statement 


•  FCS  BCT  Mission  as  a  supporting  effort  is  to 
occupy  Objective  Hula  in  order  to  secure  the  MSR 
running  from  north  to  south  through  the  objective 
and  initiate  stabilization  and  reconstruction  of  the 
urban  areas  in  Objective  Hula. 
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Notional  Mission  to  Task  Analysis 


Mission:  (U)Establish  the  Forward  Operating  Base  (98  task(s)) 

FCS  17-6-C1001  Conduct  Continuous  Risk  Management 
FCS  17-6-C1202  Operate  in  a  JIM  Environment 

FCS  17-6-S1003  Gain  and  maintain  situational  understanding  (e.g.,  the  COP)  of  the  battle  space  as  it 
applies  to  JIM  elements 

FCS  17-6-S1013  Employ  Collaborative  Networked  Fires 
FCS  17-6-S1204  Conduct  a  Tactical  Road  March 
FCS  17-6-S1402  Employ  Army  and  Joint  ISR 

FCS  17-6-S1406  Integrate  Contract/Contractor  Support  Personnel  Into  UA  Sustainment  Operations 

FCS  17-6-S2608  Transition  to  Changes  in  Focus  and  Mission  in  a  JIM  Environment 

FCS  17-6-U1004  Facilitate  Exchange  of  Information  With  JIM  Units,  Agencies  and  Representatives  at 
Appropriate  Level  using  WMI 

FCS  17-6-U1005  Coordinate  and  Perform  Intelligence,  Surveillance  and  Reconnaissance  (ISR)  in  a  JIM 

environment 

FCS  17-6-U1006  Coordinate  and  Maintain  Relevant  JIM  War  Fighter  Networks  (Warrior  Machine 

Interface) 

FCS  17-6-U1012  Participate  in  Joint  and  Multinational  Air  Space  Management 
FCS  20-2-S6203  Control  Checkpoint  (SOSO) 


Notional  FCS  BCT  Organization  for 

Combat 


•  HQ’s  BCT  (FOB  A) 

•  Group  Tiger  (FOB  B) 

-  3rd  CAB 

-  1st  CAB 

-  HHCBCT 

-  2nd  CAB 

-  BICC 

-  Team  C,  NLOS  BN 

-  NLOSBN(-) 

-  Team  C,  FSB 

-  FSB  (-) 

•  RSTA  SQDN  (FOB  C) 

Sample  Mission  Thread(s)  for  selected 

vignettes 


© 

Mission/Task/Gperation  Name 

1 

-  UA  Mission  -Conduct  Stability  and  Reconstruction  Operations 

2 

-  SPECIFIED  TASK  -  Secure  LOCs  (MAPEX  Produced  Three 

3 

Conduct  Route  Security  [DRC  Sample  Thread) 

4 

B  SPECIFIED  TASK  -  Establish  UA/CAB  FOBs  (Mapex  Produ 

5 

Execute  Task  Org  Changes  IOT  Execute  Assigned  Follow 

6 

Establish  Contacts  with  GOs:  NGOs,  and  Multinational  Su| 
Prevent  Emergency  Polulation  Crisis  Events  [Disease,  Lool 

T 

3  SPECIFIED  TASK  -  Assess  AOR 

8 

Assess  the  Infrastructure 

9 

Assess  the  Government 

1° 

Assess  Local  Leaders 

11 

Assess  Demographics 

12 

Assess  Services 

13 

Assess  Security  Needs 

14 

-  SPECIFIED  TASK  -  Normalize  the  AOR 

15 

-!  Provide  Security  Throughout  AOR 

16 

& 

Secure  the  LOG  [MAPEX  Produced  Thread) 

17 

Secure  A  Site  [MAPEX  Produced  Thread) 

18 

Repair  Vita  1  1  nfra  st  ruct  u  re 

19 

Establish  Emergency  Services  (Heath.  Sanitation.  Justice. 

20 

Establish  and  Train  Credible  Local  Officials  and  Security  F< 

21 

Integrate  NGOs  to  assist  in  Nation  Building  Activities 

22 

Establish  and  Main  PR  with  Local  Leadership  and  Media  [[ 

23 

Promote  Economic  Viability  Throughout  the  UA  AOR 

24 

-  SPECIFIED  TASK  -  Rebuild  the  AOR 

25 

Continual  Assessment  of  Ongoing  Security  Needs  and  Pro 

26 

Planning  of  Future  Projects  with  Local  Leadership 

27 

Assessing  Local  Requirements  for  Democratic  Processes 

28 

3  SPECIFIED  TASK  -  Conduct  Transfer  Authority  (TGA) 

29 

m 

Transfering  of  Authority  Where  Possible  to  Credible  Host  N 

Agfencies 

30 

S 

Prepare  the  AOR  for  Hand-Off  to  Designated  Relieving  Unit: 

31 

m 

ENDSTATE  -  Establish  Credible  Local  Government  That  Allow 
of  Coalition  Mission 

_ 

D-66 


D-36 


D-6 


D25 


D55 


DBS 


Dll  5  D145  D175  D2Q5  D235 


D265 


D295 


W  UA 
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W|  SPECIFIED  (TASK  -  Establish  |UA/CAB  |FGBs|(Mbpex  Pro 
Execute  Task  Org  Changes  IOT  Execute  Assigned  Folio 


Establish  Contacts  with  GOs,  NGOs,  arid  Multinational  Support  Organizations  IOT 


TASK  -  Normalize  the  AGR 


g  Secure  the  IOC  (MAPEX  Produced  Thread) 
1  Secure  A  Site  (MAPEX  Produced  Thread) 
g  Repair  Vital  Infrastructure 


1  Establish  Emergency  Services  (Heath, 
g  Establish  and  Train  Credible  Local 
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Sample  Mission  Thread(s)  for  selected 

vignettes 


O 

Mission/Task/Operation  Name 

Task  Ref# 

D2 

12  |  1  |2  |  3  |4  |  5  |  6  |  7  |  8  |  9  1 1 0  1 1 1  1 12  |  1  |2|3|4|5|6|7|8|9|10|11 

12  | 

1 

-  Establish  the  FOB 

FCS  7-1-U4014 

2 

3  Select/Establish  AORs 

ART  7.1.1 

3 

-  Select  FOB  location 

FCS  7-1-U4014 

4 

H  Gain  and  maintain  situational  understanding  (e.g.,  the  COP) 

FCS17-6-S1003 

of  the  battle  space  as  it  applies  to  JIM  elements 

17 

3  C2  the  UA/CAB  {Conduct  Battle  Command) 

FCS  71 -6-U2052 

18 

Establish  communications  higher  lower  adjacent  (Establish  Network 

FCS  71-5-U31 18 

rer  adjacent  (Establish  Network  Connectivity) 

Connectivity) 

19 

Maintain  Network  Connectivity 

FCS  71-5-U3119 

20 

Manage  Command  Control  Communications  and  Computers  (C4)  Gper 

FCS  71-6-S2303 

Com 

21 

Conduct  Command  Control  Communications  and  Computer  (C4)  Opera 

FCS  71-6-U2301 

Corn 

22 

% 

Control  Operations 

FCS  71-6-U2101 

23 

C4I  interoperability  in  JIM 

FCS17-6-U1006 

24 

□  Maintain  Force  Protection  {Coordinate  Force  Protection  Activities) 

FCS71-6-U2520 

25 

Conduct  Battalion  Force  Protection  Activities 

FCS  71-1-S2520 

26 

-  Protect  force  from  fires 

FCS17-6-S26Q4 

27 

Establish  Sensor  Coverage  Plan 

FCS  71-6-U2155 

28 

Control  Sensor  Platoon  Operations 

FCS6-3-U6Q17 

29 

Perform  Surveillance  and  Target  Acquisition  (Radar) 

FCS6-4-U6Q45 

lar) 

30 

Conduct  Information  Assurance 

FCS  71-5-U3124 

31 

Establish  Reponsibilities  within  the  FOB 

ART  7.5.3 

32 

V — 

Integrate  Contract/Contractor  Support  Personnel  Into  JA  Sustainmei 

FCS17-6-S1406 

Into 

33 

3  Secure  FOB  locations  {Conduct  Rear  Area  and  Base  Security  Operat 

ART  5.3. 5.4.1 

mna 

34 

Conduct  Zone/Area  Reconnaissance 

FCS  7-2-S4404 

M  Conduct  Zone/Area  Reci 

35 

Area  Surveillance 

FCS1 7-2-U401 1 

nn  Area  Surveillance 

36 

El  Assured  Mobility  (Coordinate  Assured  Mobility  Operations) 

FCS  71-6-U2525 

37 

TO 

# 

1 

3  Clear  Obstacles 

ART  5. 1.1. 2 

1  a  n~r  c  -4  -4  n  n 

JO 

39 

1 

|  Conduct  Route  Clearance  | 

Area  Access  [Conduct  Area  Clearance) 

|ART  5.1. 1.2.2 

ART  5. 1.1. 2.1 

ill  o n d o ct  Route  Clestrsirnc 
fnm  Area  Access  (Conduct  A 

:e 

rea  ( 

40 

Occupy  a  Tactical  Assembly  Area 

FCS  7-1-S4200 

THH  Occupy  a  Tactical  Asser 

ihly 

41 

Non  lethal  response  to  security  threats  (Engage  and  Control  Civilian  Pc 

FCS  7-2-S4205 

HI  Non  lethal  response  to  £ 

ecur 

42 

Secure  civilian  population  (Secure  Civilians  during  Operations  (SOSO)) 

FCS  2Q-3-S6214 

1|  Secure  civilian  populati 

on  (! 

43 

% 

Control  Networked  Fires 

FCS  71-6-U2360 

M  Control  Networked  Firer 

44 

% 

Coordinate  Early  Warning 

FCS  71-6-U2366 

HI  Coordinate  Early  Warnii 

»g 

45 

# 

3  Defend  the  FOB 

FCS7-2-S4010 

46 

-  Entry/Exit  Control 

NA 

47 

Plan  ECPs  and  Provide  Resources 

NA 

iX 

1 

. . 

k 

L 

 i 
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Implications  for  T&E  planning 


•  Need  to  develop  cross-walk  of  capabilities  to  tasks  to  sharpen  focus 
of  observation/data  collection  plan. 

•  Need  to  develop  cross-walk  of  selected  tasks  to  executors  of  tasks. 

•  Need  to  determine  interactions  needed  to  trigger  execution  of 
tasks. 

•  Need  to  maintain  coherent  mission  thread  for  vignette  being  run  in 
order  to  determine  and  trace  mission  impact  of  task  success  or 
failure  (i.e.  establish  an  audit  trail). 

•  Need  to  determine  minimum  level  of  granularity  required  for 
meaningful  execution  (test)  and  evaluation. 

•  Need  to  structure  test  support  plan  to  enable  execution, 
observation  and  data  collection  of  test  events  at  the  required  level 
of  granularity. 


S&RO  challenges 

How  do  you  determine  mission  success? 

How  do  you  determine  and  document  the  impact  of  a  single  task  on  the 
mission? 

What  are  the  unique  combinations  of  capabilities,  tasks  and  F-BCT 
components  that  must  be  tested  and  evaluated  to  determine  ESS  of  the  F- 
BCT  in  performing  S&RO  versus  other  kinds  of  missions? 

How  do  you  factor  in  augmentation  from  Echelons  above  F-BCT,  Joint, 
Coalition  and  other  capability  providers  to  the  T&E  plan? 

How  do  you  replicate  the  Operating  Environment  for  the  S&RO 
Scenario? 

-  Examples  of  S&RO  unique  conditions 

•  Forward  Operating  Base  (FOB)  based  operations  in  or  near  urban 
built  up  areas 

•  Joint,  Interagency,  Multinational  (JIM)  environment 

•  Non-linear,  Guerrilla-style  threat 

•  Fleeting  indirect  fire  threat 

•  Primarily  small  unit  (platoon/company)  size  operations 


Summary  and  Conclusion 


•  S&RO  missions  pose  unique  and  challenging  issues  to  consider  for 
T&E  planning 

•  Application  of  MMF  to  front  end  analysis  enhances  ability  to: 

-  Understand  all  aspects  of  the  problem  space  and  their 
interrelationships 

-  Bring  together  stakeholder  communities  early  for  productive  and 
effective  input,  coordination  and  review. 

-  Identify  T&E  capability  gaps  early. 


Questions/Discussion? 


- \ 

The  22nd  Annual  National 

Test  &  Evaluation  Forum 

_ _ _ / 


Sponsored  by  the  National  Defense  Industrial 
Association  Test  &  Evaluation  Division 

March  6,  2006 


Tutorial 

Putting  the  MMF  to  Work  to  Drive  Analysis  - 
Getting  to  the  “So  What”  by  Asking  the  Right 
< _ Questions  About  the  Mission _ ^ 


Held  at  The  Hyatt  Regency  Jacksonville,  Jacksonville,  FL,  6-9 

March,  2006. 
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Outline 


Missions  and  Means  Framework 
Model 

11  Fundamental  Elements:  7  levels,  4  operators 


6.  Context,  Environment  (Military,  Civil,  Physical,  etc.) 


Developed  by  Dr.  Paul  Deitz,  Technical  Director, 
U.S.  Army  Materiel  Systems  Analysis  Activity  and 
Mr.  Jack  Sheehan,  Chief  Engineer,  Future  Combat 
Systems  Combined  Test  Organization 


•  Introduction 

•  Background/MMF  Review 

•  Using  MMF  to  get  to  the  “So 
What” 

•  Practical  application  -  How  MMF 
can  be  applied  to  ask  the  right 
questions  and  organize  the 
resulting  information. 

•  A  few  examples  of  MMF 
application  to  T&E 

•  Conclusion/Summary 


Planning 

Employment 


2 


Why  Develop  New  Material  Solutions? 


A.  Because  We  Can? 

B.  Advances  in  Technology? 

C.  To  Accomplish  Operational  Missions  Under 
Changed  Operational  Environment? 


What’s  Changed? 


A.  Network-Enabled  Capabilities 

B.  Joint  Interdependence 

C.  Development  and  Fielding  of  Systems  of 
Systems 


Some  Impacts  on  T&E 

A.  How  do  you  Measure  Contribution  of  Individual  Parts  to 
Effectiveness  of  the  Whole? 

B.  How  do  you  Measure  Contribution  of  Combinations  of 
Parts  to  Effectiveness  of  the  Whole? 

C.  How  do  you  Measure  Contribution  of  Non-Lethal 
Capabilities  to  Mission  Accomplishment/Effectiveness? 

D.  How  do  you  Tie  in  Planning  for  and  Results  of 
Distributed  Test  Events  to  a  Single  Integrated  T&E 
Plan? 

E.  How  do  you  Measure  Contribution  of  Human  in  the  Loop 
and  HSI  to  Mission  Accomplishment/Effectiveness? 


ATEC’s  “mission-to-capability  trace” 


Missions 


to  accomplish 


Mission-task  decomposition  provides  the 
test  planner  with  a  powerful  tool  to 
design  the  best  test  plan  possible.  It 
stems  from  user  validated  scenarios, 
missions  and  tasks  —providing  the 
framework  for  evaluating  demonstrated 
FCS  system  and  SoS  capabilities 


comprise 


enabled  by 


DOTLPF 


* 


provides 


Answers  the 
“So  What?” 
Question 


as  supplied  by 


Materiel 


*  Doctrine,  Organization,  Training,  Materiel,  Leadership  and  Education,  Personnel,  Facilities 


Missions  &  Means  Framework  (MMF) 


•  Purpose: 

-  Methodology  for  explicitly  specifying  military  mission 

-  Quantitatively  evaluate  mission  utility  of  alternative  warfighting  DOTMLPF* 
services/products 

•  Objective:  Provide  disciplined  procedure  to  explicitly  specify  mission, 
allocate  means  and  assess  mission  accomplishment  by: 

-  Unifying  warfighter,  engineer  and  comptroller  understanding  of  missions  and 
means 

-  Accounting  for  traditional  T&E  factors  and  traditional  warfighter  expertise  factors 
that  constitute  mission  success 

-  Being  sufficiently  credible,  timely  and  affordable  to  make  hard  decisions  that  stay 
made 

-  Being  consistent,  concise,  repeatable  and  scalable 

-  Providing  disciplined  process  to  implement  SECDEF’s  transformation  guidance 
and  associated  acquisition  reform. 


*DOTMLPF  =  Doctrine,  Organization,  Training,  Materiel,  Leadership  and  Education,  Personnel  and  Facilities 


7 


Outline 


Missions  and  Means  Framework 
Model 

11  Fundamental  Elements:  7  levels,  4  operators 


6.  Context,  Environment  (Military,  Civil,  Physical,  etc.) 


Developed  by  Dr.  Paul  Deitz,  Technical  Director, 
U.S.  Army  Materiel  Systems  Analysis  Activity  and 
Mr.  Jack  Sheehan,  Chief  Engineer,  Future  Combat 
Systems  Combined  Test  Organization 


•  Introduction 

•  Background/MMF  Review 

•  Using  MMF  to  get  to  the  “So 
What” 

•  Practical  application  -  How  MMF 
can  be  applied  to  ask  the  right 
questions  and  organize  the 
resulting  information. 

•  A  few  examples  of  MMF 
application  to  T&E 

•  Conclusion/Summary 


Planning 

Employment 
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Missions  and  Means  Framework 

Model 

11  Fundamental  Elements:  7  levels,  4  operators 


6.  Context,  Environment  (Military,  Civil,  Physical,  etc.) 


7.  BLUFOR  Why  =  Purpose,  Mission  I  7.  OPFOR  Why  =  Purpose,  Mission 


il 


i=^>  Planning 


$  Employment 


Developed  by  Dr.  Paul  Deitz,  Acting  Director,  ARL 
HRED  and  Mr.  Jack  Sheehan,  Chief  Engineer, 
Future  Combat  Systems  Combined  Test 

Organization 

9 


Build  Detail 


Outline 


Missions  and  Means  Framework 
Model 

11  Fundamental  Elements:  7  levels,  4  operators 


6.  Context,  Environment  (Military,  Civil,  Physical,  etc.) 


Developed  by  Dr.  Paul  Deitz,  Technical  Director, 
U.S.  Army  Materiel  Systems  Analysis  Activity  and 
Mr.  Jack  Sheehan,  Chief  Engineer,  Future  Combat 
Systems  Combined  Test  Organization 


•  Introduction 

•  Background/MMF  Review 

•  Using  MMF  to  get  to  the  “So 
What” 

•  Practical  application  -  How  MMF 
can  be  applied  to  ask  the  right 
questions  and  organize  the 
resulting  information. 

•  A  few  examples  of  MMF 
application  to  T&E 

•  Conclusion/Summary 


Planning 

Employment 
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Applying  the  Framework  to  generate 
questions  about  the  Mission  . . . 


Framework 

Level 

Level  Description  in  Terms 
of  Military  Operations 

Questions  related  to  Test  and  Evaluation 

Level  7: 

Mission, 

Purpose 

What  is  the  Mission  Task, 
Purpose,  and  Endstate  for 
Friendly  and  threat  forces? 

What  is  the  Mission  Task,  Purpose,  and  Endstate  for 
friendly  and  threat  forces?  What  operational  scenario  is  to 
be  used  for  the  test? 

Level  6: 

Context, 

Environment 

What  are  the  conditions 
under  which  the  mission  is 
to  be  performed? 

What  are  the  environmental  factors  that  will  impact  on  the 
mission?  How  do  we  replicate  the  appropriate 
environmental  factors? 

Level  5: 

Index, 

Location/Time 

When,  in  terms  of  current 
or  future  time  period  and 
where,  in  terms  of 
geographic  location  does 
the  mission  occur? 

When  and  where  is  the  mission  to  take  place?  What  events 
(Road  to  War)  led  to  this  point?  When  and  where  will  the 
test  activities  take  place? 
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And  the  Means  that  lead  to... 


Framework 

Level 

Level  Description  in  Terms  of 
Military  Operations 

Questions  related  to  Test  and  Evaluation 

Level  4: 

Tasks, 

Operations 

What  operations  and  tasks 
must  be  performed  to 
accomplish  the  mission 

What  types  of  operations  and  what  tasks  must  the  organization 
under  test  perform  in  order  to  accomplish  the  mission?  What  is 
purpose  for  each  operation  and  task?  What  are  the  associated 
MOEs  and  MOPs?  What  conditions  are  in  place  that  could 
affect  task  performance? 

Level  3: 

Functions, 

Capabilities 

What  functions  and 
capabilities  are  required  to 
enable  successful 
accomplishment  of  task 
purpose? 

What  functions  and  capabilities  and  capabilities  are  required  to 
enable  successful  performance  of  the  operations  and  tasks  under 
identified  conditions? 

Level  2: 

Components, 

Forces 

What  equipment,  personnel 
and  forces  are  available  to 
provide  required  functions  and 
capabilities 

What  components  and  forces  are  available  to  perform  the 
mission  and  what  functions  and  capabilities  do  they  provide? 

Level  1: 

Interactions, 

Effects 

What  interactions  must  occur 
to  create  the  desired  effect. 

What  interactions/events  must  be  stimulated/simulated? 

- ts - 

T&E  Mission  Statement 


Data  Needed 

Known  Information 

Unknown  Information 

Level  2: 

Level  2: 

Level  5-7: 

Force  Listing/TOEL/TDA 

Components/F  orces 

External  operational  mission 

Level  3: 

Level  3: 

Level  4: 

Performance/Behavior 

Functions/Capabilities  to  be  delivered 

T  asks/Operations 

Function/Capability  Desc 

(based  on  O&O/ORD) 

Level  4: 

Level  3: 

Task  Lists 

Functions/Capabilities  required  in  this 
mission 

Level  5-7: 

Level  1: 

Opnl  Scenario/vignettes 

Interactions/Desired  Effects 
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T&E  Planning  and  Coordination 

Questions 


•  Does  a  suitable  scenario  exist  or  will  one  have  to  be  generated? 

•  What  are  the  most  important  capabilities  to  test  and  evaluate? 
Under  what  conditions? 

•  What  operations  and  tasks  require  the  identified  capabilities? 

•  What  components  and  forces  of  the  System  under  test  deliver 
and/or  employ  the  capabilities  identified  above? 

•  What  kind  of  LVC  environment  is  required  to  adequately  support 
observation  and  evaluation  of  the  identified  components/forces 
performing  the  identified  tasks? 

•  Do  resources  and  facilities  currently  exist  to  provide  the  required 
LVC  environment  or  will  some  have  to  be  developed? 


Implications  for  T&E  planning 


•  Need  to  develop  cross-walk  of  capabilities  to  tasks  to  sharpen  focus 
of  observation/data  collection  plan. 

•  Need  to  develop  cross-walk  of  selected  tasks  to  executors  of  tasks. 

•  Need  to  determine  interactions  needed  to  trigger  execution  of 
tasks. 

•  Need  to  maintain  coherent  mission  thread  for  vignette  being  run  in 
order  to  determine  and  trace  mission  impact  of  task  success  or 
failure  (i.e.  establish  an  audit  trail). 

•  Need  to  determine  minimum  level  of  granularity  required  for 
meaningful  execution  (test)  and  evaluation. 

•  Need  to  structure  test  support  plan  to  enable  execution, 
observation  and  data  collection  of  test  events  at  the  required  level 
of  granularity. 


15 


So  how  do  we  make  this 
work?  / 


Framework 

Level 

Fevel  Description  in  Terms  of 
Military  Operations 

Questions  related  to  Test  and  Evaluation 

Level  7: 

Mission,  Purpose 

What  is  the  Mission  Task, 
Purpose,  and  Endstate  for  F- 
BCT  and  threat  forces? 

What  is  the  Mission  Task,  Purpose,  and  Endstate  for  friendly  and 
threat  forces?  What  operational  scenario  is  to  be  used  for  the  test? 

Level  6: 

Context, 

Environment 

What  are  the  conditions  under 
which  the  mission  is  to  be 
performed? 

What  are  the  environmental  factors  that  will  impact  on  the  mission? 
How  do  we  replicate  the  appropriate  environmental  factors? 

Level  5: 

Index, 

Location/Time 

When,  in  terms  of  current  or 
future  time  period  and  where,  in 
terms  of  geographic  location 
does  the  mission  occur? 

When  and  where  is  the  mission  to  take  place?  What  events  (Road  to 
War)  led  to  this  point?  When  and  where  will  the  test  activities  take 
place? 

Level  4: 

Tasks, 

Operations 

What  operations  and  tasks  must 
be  performed  to  accomplish  the 
mission 

What  types  of  operations  and  what  tasks  must  the  organization  under 
test  perform  in  order  to  accomplish  the  mission?  What  is  purpose  for 
each  operation  and  task?  What  are  the  associated  MOEs  and  MOPs? 
What  conditions  are  in  place  that  could  affect  task  performance? 

Level  3: 

Functions, 

Capabilities 

What  functions  and  capabilities 
are  required  to  enable  successful 
accomplishment  of  task 
purpose? 

What  functions  and  capabilities  and  capabilities  are  required  to 
enable  successful  performance  of  the  operations  and  tasks  under 
identified  conditions? 

Level  2: 

Components, 

Forces 

What  equipment,  personnel  and 
forces  are  available  to  provide 
required  functions  and 
capabilities 

What  components  and  forces  are  available  to  perform  the  mission 
and  what  functions  and  capabilities  do  they  provide? 

Level  1: 

Interactions, 

Effects 

What  interactions  must  occur  to 
create  the  desired  effect. 

What  interactions/e  vents  must  be  stimulated/simulated? 

Outline 


Missions  and  Means  Framework 
Model 

11  Fundamental  Elements:  7  levels,  4  operators 


6.  Context,  Environment  (Military,  Civil,  Physical,  etc.) 


Developed  by  Dr.  Paul  Deitz,  Technical  Director, 
U.S.  Army  Materiel  Systems  Analysis  Activity  and 
Mr.  Jack  Sheehan,  Chief  Engineer,  Future  Combat 
Systems  Combined  Test  Organization 


•  Introduction 

•  Background/MMF  Review 

•  Using  MMF  to  get  to  the  “So 
What” 

•  Practical  application  -  How  MMF 
can  be  applied  to  ask  the  right 
questions  and  organize  the 
resulting  information. 

•  A  few  examples  of  MMF 
application  to  T&E 

•  Conclusion/Summary 


Planning 

Employment 
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MMF  Builds  on  Doctrinal  Processes 


•  MDMP 

-  Mission  Analysis 

-  COA  Development 

-  Wargaming 

•  Operations 

•  Training  &  Readiness 

-  JTS  Phases 

-  JMETL/METL 
Development  and 
Assessment 


A  FM  5-0 

i’'  |FM  101-SI 

Army 

Planning  and  Orders 

^  Production 

JAHUMWaOCO 

MFUKKrARTtn? 

DEPARTMENT  OF  THE  ARM* 

n 
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The  MDMP  -  Its  About  the  Mission 


FM  5-0 

{FM  101-5) 


Army 

Planning  and  Orders 
Production 


JANUARY  £0C3 

DaminxH  HEriucm 

HEADQUARTERS 
DEPARTMENT  OF  THE  ARMY 


Input 

-  Mission  received  from  higher  HQ's  or 
deduced  by  the  commander  and  staff 


-  Higher  HQ  order/plan/ 1 PB 

-  Staff  estimates 

-  Facts  &  assumptions 


*  Restated  mission 

*  Cdr’s  intent  and  planning  guidance 

-  GCIR 

-  Staff  estimates  and  products 

-  Enemy  CQAs 


*  Refined  Cdr’s  planning  guidance 

-  Enemy  CQAs 

-  Friendly  COAs 

-  StafffBOS  CQAs 

*  War-game  results 

*  Established  criteria  for  comparison 


*  Decision  matrix 


-  Approved  COA 


Receipt  of  Mission _ 

~WARNO  | 

Mission  Analysis 


COA  Development 

WARNO 

COA  Analysis 
(War-Game) 

COA  Comparison 

COA  Approval 

WARNO  1 

Orders  Production 

Output 

)  Cdr’s  initial  guidance 

■  WAR  MO  to  staff 

*  Initial  IPB  products 
"1  Restated  mission 
^  Crd's  intent 

"1  Cdr’s  planning  guidance 
^  Initial  GCIR 

^  Updated  staff  estimates/products 

*  Preliminary  movement 

*  Initial  1SR  Plan 

*  Updated  staff  estimates 

*  COA  statements  and 
sketches 

'1  Refine  Crd's  planning  guidance 

■  War-game  results 

*  Decision  support  templates 

*  Task  Organization 

*  Mission  to  subordinate  units 


*  Decision  matrix 


)  Approve  COA 

)  Refine  Crd’s  i  ntent  and  gui  dance 
"l  Refine  GCIR 


•  OPLAN/OPROD 


Note  1:  A  star  depicts  commander 
activities  and  or  decisions. 

Note  2:  Rehearsals  and  back-briefs 
occur  during  preparation  and  ensure  an 
orderly  transition  between  planning  and 
execution. 


Preparation 


Execution 


Assess 


Figure  3-1.  The  Military  Decision  Making  Process 


Mission  /  Operation  /  Task 


MISSION 

REQUIREMENTS 


M 


TASK 

PERFORMANCE 


We  measure  Task  Performance 

-  Task  to  Operation  (Small  to  Big) 


We  measure  Mission  Accomplishment 

-  Are  we  getting  the  job  done  (MOEs) 


Mission  to  Task  Decomposition  - 
Establishes  Vertical  Linkage. . . 


*Note  Detachment  METLs  may  become  battle  tasks  for  higher  level  commands 
(Battalion  through  Theater  Army) 


Fignre3-10.  Relationships  Between  Mission,  METL,  ;md  Battle  Task 


. . .  and  Horizontal  Linkage  between 

subordinate  units . . . 


Corps  Mission  Esse 
Division  Mission  Es 

intial  Task:  Conduct  Shaping  Op 
isential  Task:  Conduct  Deliberat 

>erations 
e  Attack 

BATTLE  TASK 

UNIT  TASK  SELECTED 

BATTLEFIELD  OPERATING 
SYSTEMS 

Conduct  Deliberate  Attack 

1st  and  2d  Brigade 

Maneuver 

Conduct  Deep  Attack 

Aviation  Brigade 

Maneuver 

Provide  Fire  Support  for 

Combat  Operations 

DIVARTY 

Fire  Support 

Reduce  Complex  Obstacles  for 

Division  Deliberate  Attack 

Engineer  Brigade 

Mobility  /  Counter-mobility 
!  Survivability 

Provide  and  Manage  DS  Combat 

Service  Support 

DISCOM 

Combat  Service  Support 

Conduct  Intelligence  and  Electronic 

Attack  Operations 

Ml  Battalion 

Intelligence 

IOM  Tactical  Communications  Networks 
(C4I)  for  the  Division 

Signal  Battalion 

Command  and  Control 

Coordinate  Air  Defense  C3I  and 

Directed  Early  Warning 

Air  Defense  Artillery  Battalion 

Air  Defense 

Figure  3-9.  Example  List  of  Division  MSC  and 

Separate  ButLdhm  Battle  Tasks  tliu 

t  Support  a  Mission  Essential  Task 

•  •  • 


and  Horizontal  Linkage  between  Staff 

Elements 


Element/Section 

Staff  Mission  Essential  Task 

SI 

Provide  Personnel  Services  Support 

S2 

Conduct  Intelligence  Preparation  of  the  Battlefield 

Develop  an  R&S  Plan 

S3 

Plan  Operations  using  MDMP 

Prepare  OPLAN/OPORD 

Develop  Task  Organization  Annex  and  Ops  Overlay 

S4 

Develop  Logistics  Estimate 

S5 

Coordinate  Host  Nation  Support 

SG 

Establish  Information  Network 

DECOORD 

Coordinate  Fire  Support 

ENGR 

Conduct  M/CM-  S  Operations 

ADA  LNO 

Conduct  ADA  Planning  and  Coordination 

AVN  LNO 

Conduct  Army  Aviation  Planning  and  Coordination 

USAFALO 

Conduct  A2C2  Management  and  Control  Activities 

Figure  3-7.  Example 

?  of  Brigade  Staff  METL 
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Vertical/Horizontal  Mission  to  Task 
Linkage  -  Joint  Perspective 


Sample  JMET 


JMET:  Coordinate  Theater-wide  Information 
Operations  (IO)  (ST  5.5) 

Organization:  J-3 

*  Conditions 

-  C  2. 3.2.3  Flexibility  of  Warfare  Style  (Flexible) 

-  C  2.4.4  Theater  Intelligence  Organizations 
(Mature) 

*  Standards 

-  90  percent  of  subordinate  plans  have  integrated 

C2W  efforts 

-  10  davs  to  achieve  information  superiority 


Supporting  Tasks  -  Identify  Theater  Issues  and 
Threats  (ST  2.4.1.11 


-  Conditions 

■  C  2.3.2  Military  Style  (Predictable) 

-  Standards 

_ ■  1  0  hen rs  nr  less  to  identify  enemy  COO 

Command-linked  Tasks  -  Support  National  and 
JTF  Surveillance  Reconnaissance  Requirements 
(ST  2.2.2) 


-  CuriLllLluris - 

■  C  1 .3.2  Visibility  (High) 

-  Standards 

■  90  percent  of  JQA  has  surveillance 
coverage 


Vertical  Linkage 


Horizontal  Linkage 
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Sample  METL  -  Army  Brigade 


Alert  and  Deploy  the  Brigade 

*  Draw  and  upload  basic/operational  loads  *  Move  by  road  or  rail  to  APOE  or  SPOE 

*  Conduct  soldier  readiness  administrative  /  *  Upload  equipment  at  APOE  or  SPOE 

logistic  preparation  for  overseas  movement 

*  Deploy  advance  parties  or  liaison  officers 

Conduct  Attack 

*  Attack  a  Moving  Enemy 

*  Attack  a  Stationary  Enemy 

*  Movement  to  Contact 

Conduct  Defense 

*  Conduct  a  Mobile  Defense 

*  Conduct  an  Area  Defense 

Conduct  Support  Opns 

Conduct  Stability  Opns 

•  Domestic  Support  Operations 

*  Foreign  Humanitarian  Assistance 

*  Peacekeeping  Operations 

*  Combat  Terrorism 

*  Support  Counter- Drug  Operations 

Conduct  Sustainment  Operations 

*  Provide  medical  treatment  and  evacuation 
(air  and  ground) 

*  Move  by  air  surface  transportation 

*  Manage  Terrain 

*  Recover  and  evacuate  disabled  equipment 

*  Control  reconstitution  of  subordinate  units 

*  Conduct  mortuary  affairs  operations 

Figure  3-2.  Example  of  Brigade  Tasks  Derived  from  Wartime  Operational  Plans  and  External  Guidance 
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Putting  it  All  Together  -  COCOM  to  Squadron  Linkage  example 


Place  forces  in  position  to 
begin  major  operations  in 
order  to  achieve  strategic 
and  operational  objectives. 


JSCP 


Supported 

CDR 


Mission  Analysis 


JMET:  ST  1.3 

Conduct  Thtr  Strat  Mnvr 
&  Force  Positioning 


Command  Linked  Task: 

Supporting  Task: 

ST  1.1 

ST  6.2 

Conduct  Intratheater  Strat  Dplmt 

Coordinate  protection  for  theater 

Supporting 

OPLAN 


Support  the 
Combined  Cdr  with 
forces  from  outside 
the  theater  of 
operations. 


Combatant 

CDR 

s... - -  / 

Supported  CDR 


Prevent  enemy  forces 
from  disrupting  flow  of 
forces  within  the  theater 
of  operation. 


[^Mission  Analysisj] 


[^Mission  Analysis]] 


Supporting 

OPLANs 


MET:  ST  7.1.2 

Determine  Deployment 
Reqmts 


Command  Linked  Task: 

Supporting  Task: 

SN  1.2 

ST  1.1.4 

Conduct  Dplymt  /  Redeplymt 

Provide  C2  of  Deploying  Units 

Supporting 

OPLANs 


Maintain  auth.  & 
dir.  of  deploying 
USAF  units  until 
handoff  to  ACC. 


Eliminate 

>  „  ^  potential  enemy 

[^Mission  Analysis^^-~~~|  threats  to  mvmnt. 


Maintain  freedom  of 
mvmnt  into  theater 
of  operations 


Attk  specific  tgts  V  CMission 
before  they  can 


MET:  AFT  1.1.2 

Train  counterair  forces 


affect  ops 


MMF  Formal  Process  Diagram 


6  Carted;  Bivirormert  (Mltery.  Civil,  Fh/sicsl,  etc) 


7.BJUPCRWy=PLipcGes  Msaon 


7.  CH-CRV\fy=Ripose;  Msaon 


% 


*  q, 

n  O  rVrmmor#c 


2  Ccrrpcnerts, 
Forces 


5  Incbc  Location 
&Time 


Tf 

Jk 

k1> 


ticns 

'*X°“ 


2  Corporals 
Forces 


*/ 


% 


XT 


END 

(Mission 
ccomplished) 


Level  7 


Mission/OPLAN/ 
Training  Obj. 

Step  1 


Levels  7-4 


Conduct 

CDR’s 

Mission 

Analysis 

Step  2 


Specify  The  Mission 


Lvl  4  Influenced  by  Lvl  6 


Level  3 


O3.4S 


Associate  Tasks 
With  Required 
Capabilities 

Step  7 


Allocate  Means 


Available 

Components 


Conduct 

Capabilities 

Analysis 

Step  6 


Associate  Tasks 
With  Conditions 
&  Measures/ 
Standards 

^tep 


Z Joint/Service  / 
Task  List  / 


Lvl  4  Influenced  by  Lvl  5 


Associate  Task 
Interrelationships 
&  Time-Phasing 

Step  5 


Level  4 


Build  Mission 
Task  List 

Step  3 


fo2.3s 


Level  2 


Associate 
Components 
With  Required 
Capabilities 
Step  8 


Synthesis 


Associate 
Components 
With  Tasks 
Step  9 


No 


No 


Yes 


End-state 

Achieved? 


Yes 


Purpose 

Achieved? 


Employment 

O^E 

Assess 
Effectiveness 

Step  1 5 


O3.4E 

Compare 

Task  Req.  Capability 
(7)  To  Res.  Capability 
(13) 

Step  14 

Level  4 


Assess/Determ  i  ne 
Residual  Capability 

Step  13 

Level  3 


O  F 

U1-2C 

LEVEL  1y 

Observe/Record 

Comp/Forces 

Execute  COA 

^ —  State  Change 

< — 

(LVC) 

Step  1 2 

Step  1 1 

Level  2 

Mission/OPLAN/ 
Training  Obj. 

Step  1 


Need  mission  types  and 

Scenario/vignettes  from 
Requirements 
Documents  and/or  User 
Representative 


•  What  Mission(s)  is/are  to  be  performed  by  the  object  under  test? 

•  Doctrinal  mission  type(s)  (I.e.  Stability  Ops  mission(s)) 

•  Doctrinal  references  to  be  used  (Concepts,  FM’s,  etc.) 

•  What  Operating  Environment  is  to  be  used  to  set  the  conditions  for 
performance  of  the  mission?  What  is  the  Road  to  War  that  established  the 
Operational  context  for  each  each  test  vignette?  What  approved  scenario  has 
been  selected  to  provide  the  appropriate  environment  and  context? 

•  Scenario/vignette  documentation  (MDMP  products,  maps  or  map  references,  COA 
statements  and  sketches,  plans  and  orders,  annexes,  graphics,  etc.). 

•  Including  selected  OPLAN/OPORD  products  at  least  two  levels  up. 


UA  Vignette  /  Scenario  Relationship 


Scenarios Caspian 


Balkan 


SW  Asia 


NE  Asia 


Bluefor 

'icinettes 

Opfor 

Bluefor 

Opfor 

Bluefor 

Opfor 

Bluefor 

Opfor 

Vig  #1  Entry 
Operations 

< 

^“^storybo* 

uct? ' 

s> 

ards  et£Y^ 

Vig  #2  CA  Ops  in 
major  urban  areas 

Vig  #3  Mounted 

Exploitation/Pursuit 

Ops 

Vig  #4  Rapid 

Advance  to  enemy 
COG 

Vig  #5  AASLT  in 
mountain  terrain 

Vig  #6  Dismt  Ops  to 
conduct  raid  on 
decisive  point  in 
jungle 
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Conduct 

Mission 

Analysis 

Step  2 


\ccompnsnea) 

Step  8 

“1 

y  u4.,e 

O3.4E 

02.3E  o1.2e 

LEVEL  T 

Yes  A 

Assess 

Compare 

Task  Req.  Capability 

Assess/Determine  Observe/Record 

Execute  COA 

Achieved? 

Purpose 

Achieved? 

Effectiveness 

Step  15 

(7)  To  Res.  Capability 
(13) 

Step  14 

Residual  Capability  . Comp/Forces 

SteD  13  state  Change 

Step  12 

Step  1 1 

•  Use  the  appropriate  (Joint/  lervice  Specific)  doctrinal  planning  and  decision  making 
process  to: 

•  Analyze  the  assigned  or  perceived  mission  and  determine/document  the  specified 
and  implied  tasks  (with  their  associated  purpose)  required  to  accomplish  the 
mission. 

•  Determine  and  document  facts,  limitations,  assumptions  and  constraints. 

•  Describe  Operating  Environment  and  Impact  on  Operations 

•  Describe  Current  Situation  and  Events  leading  up  to  it. 

•  Describe  Threat  Situation,  Mission,  Intent,  Vulnerabilities,  etc. 


MDMP 


r 


FM  5-0 

{FM  101-5) 


Army 

Planning  and  Orders 
Production 


JANUARY  2GC5 


Input 


*  Mission  received  from  higher  HQ's  or 
deduced  by  the  commander  and  staff 


■  Higher  HQ  order/plan/ 1 PB 

■  Staff  estimates 

■  Facts  &  assumptions 


■  Restated  mission 

■  Cdr's  intent  and  planning  guidance 

■  GGIR 

■  Staff  estimates  and  products 

■  Enemy  CQAs 

■  Refined  Cdr’s  planning  guidance 

■  Enemy  COAs 

■  Friendly  COAs 

■  Staff/BOS  COAs 

■  War-game  results 

■  Established  criteria  for  comparison 


■  Decision  matrix 


-  Approved  COA 


Receipt  of  Mission 


WARNO 


Mission  Analysis 


COA  Development 

WARNO  I 

COA  Analysis 
(War-Game) 

COA  Comparison 


COA  Approval 


WARNO 


Orders  Production 


Output 


\  Cdr’s  initial  guidance 

-  WARNO  to  staff 

*  Initial  IPB  products 
"1  Restated  mission 
^  Crd's  intent 

"1  Cdr’s  planning  guidance 
^  Initial  CCIR 

^  Updated  staff  estimates/products 

*  Preliminary  movement 

-  Initial  1SR  Plan 


*  Updated  staff  estimates 

*  COA  statements  and 
sketches 

'1  Refine  Crd's  planning  guidance 


*  War-game  results 

*  Decision  support  templates 

*  Task  Organization 

*  Mission  to  subordinate  units 


►  Decision  matrix 


\  Approve  COA 

^  Refine  Crd’s  i  ntent  and  gui  dance 
"l  Refine  CCIR 


■  OFLAN/OPROD 


Note  1:  A  star  depicts  commander 
activities  and  or  decisions. 

Note  2:  Rehearsals  and  back-briefs 
occur  during  preparation  and  ensure  an 
orderly  transition  between  planning  and 
execution. 


Preparation 


Assess 


Execution 


Figure  3-1.  The  Military  Decision  Making  Process 


l 


Mission  Analysis  Process. . . 


•  Mission  received  from  higher  HQ's  or 
deduced  by  the  commander  and  staff 


•  Higher  HQ  order/pJan/IPB 

•  Staff  estimates 

•  Facts  &  assumptions 


•  Restated  mission 

•  Cdr's  intent  and  planning  guidance 

•  CCIR 

•  Staff  estimates  and  products 

•  Enemy  COAs 

•  Refined  Cdr’s  planning  guidance 

•  Enemy  COAs 

•  Friendly  COAs 

•  Staff/BOS  COAs 

•  War-game  results 

•  Established  criteria  for  comparison 


•  Approved  COA 


Receipt  of  Mission 

f~WAR^ 

Mission  Analysis 


COA  Development 


COA  Analysis 
(War-Game) 

COA  Comparison 
COA  Approval 

|~WARNO 

Orders  Production 


Output  ^ 

•  Initial  IPB  products 
"l  Restated  mission 
'I  Crd’s  intent 

't  Cdr’s  planning  guidance 
'I  Initial  CCIR 

Updated  staff  estimates/products 

•  Preliminary  movement 

•  Initial  ISR  Plan 

•  Updated  staff  estimates 
w  •  COA  statements  and 
fetches 

^Refine  Crd's  planning  guidance 


•  War-Xme  results 

•  DecisiXsupport  templates 

•  Task  Organization 

•  Mission  to  subordinate  units 


•  Decision  matnx 


A  Approve  COA  X 
'I  Refine  Crd’s  intent  and  c 
^  Refine  CCIR 


•  OPLANyOPROD 


Input 

Higher  HQ  plan 
or  order 
Higher  HQ  IPB 
Updated  staff 
estimates 
Initial  cdfs 
guidance 


Process 

Analyze  the  higher  HQ  order 
Perform  initial  IPB 
Determine  specified,  implied,  and 
essential  tasks 
Review  available  assets 
Determine  constraints 
Identify  critical  facts  and 
assumptions 
Perform  risk  assessment 
Determine  IR  and  initial  PIR,  FFIR, 
and  CCIR 

Determine  the  initial  ISR  plan 
Update  operational  timelines 
Write  the  unit  mission  statement 
Deliver  the  mission  analysis  briefing 
Approve  the  unit  mission  statement 
Develop  the  initial  commander's 
intent 

Issue  Cdr's  planning  guidance 
issue  a  warning  order 
Review  facts  and  assumptions 


Output  \ 

Mission  analysis  \ 
briefing  \ 

Initial  IPB 

•  Enemy 
SITTEMPs 

•  m:cqo 

•  HVTs 

Restated  mission 
Initial  Cdr's  intent 
Commanders  > 

planning  guidance  j 
Warning  order  / 


Note  1  A  star  depicts  commander 
activities  and  or  decisions. 

Note  2:  Rehearsals  and  back-briefs 
occur  during  preparation  and  ensure  an 
orderly  transition  between  planning  and 
execution. 


Assess 


Figure  3-5.  Mission  Analysis 


Figure  3-1.  The  Military  Decision  Making  Process 


Yields  MMF  Level  7-5  data,  list  of 
Mission  Tasks  (big  part  of  Level  4),  and 
initial  Force  Listing  (Level  2) 


Figure  3-5.  Mission  Analysis 


3 


Conditions 


•  Conditions  are  variables  of  the  operational 
environment  or  situation  that  may  affect 
performance. 


•  Environment  is  the  immediate  situation  or 
circumstance  in  which  tasks  must  be  performed. 


•  Conditions  impact  the  ability  to  perform  a  task  (e.g., 
security  of  lines  of  communication  impact  the  ability 
to  sustain  joint  task  force  operations). 
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Conditions 


1.1 

Land 

C  2.1 

Mission 

1.2 

Sea 

C  2.2 

Forces 

1.3 

Air 

C  2.3 

Command,  Control  & 

1.4 

Space 

Communications  Related  Conditions 

C  2.4  Intelligence  Related  Conditions 
C  2.5  Deployment,  Movement,  and  Maneuver 


Related  Conditions 
C  2.6  Firepower  Related  Conditions 
C  2.7  Protection  Related  Conditions 
C  2.8  Sustainment  Related  Conditions 
C  2.9  Threat  Related  Conditions 
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Joint/Service 
Task  List 


•  Determine/select  the  source  of  tasks  to  be  used  to  describe  mission  activities  for  the 
object  under  test. 

•  Examples:  UJTL,  AUTL,  UNTL,  MCTL,  IATL,  FCS  Single  Integrated  Task 
List  (SITL),  Soldier  Training  Plan  (STP),  Soldiers  Manual  of  Common  Tasks 
(SMCT),  etc. 

•  Ensure  the  selected  task  list  is  current  and  contained  in  the  data  base  to  be 
accessed  by  the  task  management  tool  (I.e.  JTIMS,  DTMS,  ATIA,  etc.) 

•  Determine/select  an  alternative  source  for  acceptable  substitute  tasks  in  the  event 
primary  source  does  not  have  an  adequate  task  description  for  a  particular  activity* 


•  Tasks  should  come  from  authoritative  (i.e.  approved)  source.  Recommend  you 
avoid  using  newly  constructed  and  not  yet  approved  tasks. 
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Authoritative  Task  Lists 


CJCSK  3500.&4& 
1  Aifiit  2005. 


UNIVERSAL  JOINT  TASK  LIST 
(UJTL| 


JOINT  STAFF 

WASHINGTON  D.C;  20318 


* 

FWI  MS 

* 

THE  ARMY 
UNIVERSALTASK 

LIST 

* 

AUGUST  2003 

IT 

HEADQUARTERS 

DEPARTMENT  OF  THE  ARWV 

•  Several  authoritative 
sources  available  to 
describe  elements  of 
military  operations 

•  Authoritative  task  lists 
are  easily  stored  and 
maintained  in  data 
bases  for  access  and 
manipulation 
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Build  Mission 
Task  List 


•  Convert  the  mission  (specified,  implied)  tasks  from  step  2  into  tasks  from  the  selected 
authoritative  task  list. 

•Use  doctrinal  references  for  the  types  of  missions  and  operations  to  be  conducted  to 
identify  supporting  tasks  by  functional  area  or  Battlefield  Operating  System,  or  some  similar 
organizing  principle. 

•  Approved  doctrinal  templates  of  tasks  already  exist  for  many  doctrinal  mission  types  and  should 
be  used  as  a  starting  point  wherever  possible.  Potential  sources  include  the  UJTL,  Navy  NMETL’s, 
and  Army  Mission  Training  Plans. 

•  Templates  will  likely  have  to  be  modified  to  reflect  specific  aspects  of  the  operational  scenario. 

•  Capture,  document  and  organize  the  resulting  list  of  tasks  into  a  data  base  mission  file  (see 
next  slide) 

•  This  should  be  done  in  conjunction  with  Step  2,  Mission  Analysis  or  as  closely 
thereafter  as  possible 


Basis  for  Conversion  to  Authoritative  Task 

Language 

JMETL  Development  Guidance  (CJCSM  3500. 03A,  1  Sept  2002) 

Perform  Mission  Analysis  to  determine  specified  and  implied  tasks. 

Select  appropriate  UJTL  tasks  in  JTIMS  that  are  descriptive  of  specified  and  implied  tasks.  Result  is  list 
of  Mission  Tasks  which  are  capabilities  associated  with  each  assigned  mission 

Determine  which  tasks  are  essential  by  applying  commander-approved  essentiality  criteria  for  each 
mission  to  each  mission  task. 

Assign  responsible  organizations,  conditions  and  standards  for  that  mission.  Conditions  are  selected  if 
they  affect  task  performance.  Standards  relate  to  task  performance  under  mission  conditions. 

Assign  supporting  and  command  linked  tasks.  Completed  JMETS  for  all  assigned  missions  with 
supporting  and  command  linked  tasks  constitute  a  command’s  JMETL.  Supporting  tasks  =  Mission  for 
subordinate  organization 
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Sample  Conversion  of  specified,  implied  tasks  to  authoritative  task  list  tasks 


33 


Network 

Diagram 


Mission  to  task  analysis  results  —  Specified  and 

Specified  and  implied  tasks  converted  to  tasks 
from  authoritative  task  lists 

Task  Ref.  Number 

0 

implied  tasks 

1  1 

□  Establish  the  FOB  V  V 

— ESfaBlish  Forward^pwaling  Base 

FCS  7-1 -U4014 

1  2 

B  Select/Establish  AORs^v 

ish  and  Condujpr^ommand  Post  Operations  to  Support  Tactical  Oper 

ART  7.1.1 

1  3 

B  Select  FOB  location 

EstablisJ^erfward  Operating  Base 

FCS  7-1 -U4014 

EGain  and  maintain  situational  unHeTUtandiiig 
of  the  battle  space  as  it  applies  to  JIM  elements 

Gain  and  mainTgunsituational  understanding  (e.g.,  the  COP)  of  the 
battle  space  as  it  applies  to  JIM  elements  which  may  affect  the  Unit 

FCS17-G-S10D3 

E  C2  the  UA/CAB  {Conduct  Battle  Command) 

C2  the  UA/CAB  (Conduct  Battle  Command) 

FCS  71 -6 -U  20  52 

1  24 

B  Maintain  Force  Protection  (Coordinate  Force  Protection  Activities) 

Coordinate  Force  Protection  Activities 

FCS71-G-U2520 

j  25 

Conduct  Battalion  Force  Protection  Activities 

Conduct  Battalion  Force  Protection  Activities 

FCS  71-1-S2520 

|  26 

B  Protect  force  from  fires 

Conduct  Defense  Maximizing  Collaborative  Networked  Fires 

FCS17-6-S2604 

1  27 

Establish  Sensor  Coverage  Plan 

Integrate  Sensor  Operations 

FCS  71-6-U2155 

;  28 

Control  Sensor  Platoon  Operations 

Control  Sensor  Platoon  Operations 

FCS6-3-U6017 

29 

Perform  Surveillance  and  Target  Acquisition  (Radar) 

Perform  Surveillance  and  Target  Acquisition  (Radar) 

FCS64-UGG45 

30 

Conduct  Information  Assurance 

Conduct  Information  Assurance  (IA) 

FCS  71-5-U3124 

31 

Establish  Reponsibilities  within  the  FOB 

Task  Organize  For  Operations 

ART  7.5.3 

32 

Integrate  Contract/Contractor  Support  Personnel  Into  LIA  Sustainmei  igrate  Contract/Contractor  Support  Personnel  Into  UA  Sustainment  Operatic 

FCS17-6-S1406 

33 

B  Secure  FOB  locations  {Conduct  Rear  Area  and  Base  Security  Operat 

Conduct  Rear  Area  and  Base  Security  Operations 

ART  5. 3. 5.4.1 

34 

Conduct  Zone/Area  Reconnaissance 

Conduct  Zone/Area  Reconnaissance 

FCS  7-2-S4404 

35 

Area  Sun/eillance 

Conduct  Area  Surveillance 

FCS17-2-U401 1 

36 

B  Assured  Mobility  (Coordinate  Assured  Mobility  Operations) 

Coordinate  Assured  Mobility  Operations 

FCS  71 -6 -U 25 25 

37 

B  Clear  Obstacles 

Clear  Obstacles 

ART  5. 1,1. 2 

j  38 

Conduct  Route  Clearance 

Conduct  Route  Clearance 

ART  5.1  1.2.2 

39 

Area  Access  (Conduct  Area  Clearance) 

Conduct  Area  Clearance 

ART  5. 1.1. 2.1 

40 

Occupy  a  Tactical  Assembly  Area 

Occupy  a  Tactical  Assembly  Area 

FCS  7-1-S4200 

;  41 

Non  lethal  response  to  security  threats  (Engage  and  Control  Civilian  Pc 

Engage  and  Control  Civilian  Populations 

FCS  7-2-S4205 

42 

Secure  civilian  population  (Secure  Civilians  during  Operations  (SOSO)} 

Secure  Civilians  During  Operations  (SOSO) 

FCS  20-3-36214 

43 

% 

Control  Networked  Fires 

Control  Networked  Fires 

FCS  71-5-U2360 

44 

% 

Coordinate  Early  Warning 

Coordinate  Early  Warning 

FCS  71-6-U2366 

45 

B  Defend  the  FOB 

Conduct  Area  Security  Operations. 

FCS7-2-S4010 

46 

B  Entry/Exit  Control 

HA 

47 

% 

Plan  ECPs  and  Provide  Resources 

No  Match 

NA 

48 

% 

Establish  the  ECP 

Control  Checkpoint(s)  (SOSO) 

FCS  20-2-C6203 

49 

Operate  an  ECP 

Operate  Checkpoint  (SOSO) 

FCS20-3-S6207 

50 

B  Prepare  Protective  Positions 

Prepare  Protective  Positions 

ART  5.3.1. 3 

]  51 

Employ  Protective  Obstacles 

Employ  Protective  Obstacle(s) 

FCS  7-3-U4025 

52 

#■% 

Construct  Protective  Earth  Wall  Berms  and  Revetments 

Construct  Protective  Earth  Walls,  Berms.,  and  Revetments 

ART  5.3. 1.3.1 

53 

Construct  Vehicle  Information  System  Equipment,  and 

Material  Protective  Positions 

Construct  Vehicle,  information  System  Equipment,  and  Material 
Protective  Position 

ART  5.3. 1.3.2 

Tracking 

Gantt 


Resource 

Usage 


A 


Ready 


NUM 


BCRL 


3VR 


Sample  JTIMS  Mission  Task  File  (screen  shot) 


Mission:  SLAD  Demo  MCS  CO  CAB2  -  Conduct  an  Attack 


□IK 


j  Mission  |  Conditions  j  Doctrine  *  |  Guidance  MTAs  *  | 


Z3  SLAD  Demo  MCS  CO  CAB2  -  Conduct  an  Attack 
9  C3  LSI  A2.7  EXECUTE  TACTICAL  OPERATIONS  [ART  7.6] 

9  Supporting  Tasks 

9  Cl  LSI  A2.1  EXERCISE  BATTLE  COMMAND 
9  C3  Supporting  Tasks 

9  C3  LSI  A2.3  MANAGE  TACTICAL  INFORMATION  [ART  7.2] 

9  IZ3  Supporting  Tasks 

©-  C3  LSI  A2.3.1  COLLECT  RELEVANT  INFORMATION  [ART  7.2.1] 

©-  a  LSI  A2.3.2  PROCESS  RELEVANT  INFORMATION  TO  CREATE  A  COMMON  OPE  : 
©-  C3  LSI  A2.3.3  DISPLAY  A  COP  TAILORED  TO  USER  NEEDS  IN  EASILY  UNDERST(  j 
®-  C3  LSI  A2.3.4  STORE  RELEVANT  INFORMATION  [ART  7.2.4] 

Hi  Command-Linked  Tasks 

9  C3  LSI  A2.4  ASSESS  TACTICAL  SITUATION  AND  OPERATIONS  [ART  7.3] 

9  C3  Supporting  Tasks 

©-  C3  LSI  A2.4.1  MONITOR  SITUATION  OR  PROGRESS  OF  THE  OPERATION  [ART  7.; 
©-  Cl  LSI  A2.4.3  PROVIDE  COMBAT  ASSESSMENT  [ART  7.3.3] 

Hi  Command-Linked  Tasks 
n  Command-Linked  Tasks 

9  C3  LSI  A5.1 .3  EMPLOY  FIRES  TO  INFLUENCE  THE  WILL  f  DESTROY  OR  SUPPRESS  ENEMY  FO  \ 
9  Cl3  Supporting  Tasks 

9  C3  LSI  A5.1.3.1  CONDUCT  LETHAL  FIRE  SUPPORT  [ART  3.3.1] 

9  (Z3  Supporting  Tasks 

©-  C3  LSI  A5.1 .2  DETECT  AND  LOCATE  SURFACE  TARGETS  [ART  3.2] 

©-  C3  LSI  A5.1 .3.1 .1  CONDUCT  AIR  -TO  -  SURFACE  ATTACK  [ART  3.3.1 .2] 

9  C3  LSI  A5.1 .3.1 .2  CONDUCT  SURFACE -TO  -  SURFACE  ATTACK  [ART  3.3.1 .1] 

9  C3  Supporting  Tasks 

©-  C3  LSI  A5.1 .3.1 .2.1  CONDUCT  NLOS  BATTERY  OPERATIONS 
9  C3  Command-Linked  Tasks 

©-  C3  LSI  X . EMPLOY  NETWORKED  FIRES  TO  ACCESS  EXTERNAL 

Hi  Command-Linked  Tasks 
H|  Command-Linked  Tasks 
o  C3  LSI  A7  Perform  Maneuver  Sustainment 
9  C3  ART  1 .0  The  Intelligence  Battlefield  Operating  System 
9  (Z3  Supporting  Tasks 

9  C3  LSI  A4.2  COLLECT  INTELLIGENCE  [ART  1 .2] 

9  Supporting  Tasks 

©-  C3  LSI  A4.2.1  CON  DUCT  TACTICAL  RECONNAISSANCE  [ART  1 .2.1] 

©-  C3  LSI  A4.2.2  CONDUCT  SURVEILLANCE  [ART  1 .2.2] 

H  Command-Linked  Tasks 

©-  LSI  A4.3.1  PERFORM  SITUATION  DEVELOPMENT  [ART  1 .3.1] 

O  LSI  A4.3.2  Perform  Target  Development  and  Support  to  Targeting  [ART  1 .3.2] 

Hi  Command-Linked  Tasks 

9  C3  ART  2.0  The  Maneuver  Battlefield  Operating  System 
9  C3  Supporting  Tasks 

O  C3  LSI  A1 .2  CONDUCT  TACTICAL  MANEUVER  [ART  2.2] 

LSI  A1. 3  CONDUCT  TACTICAL  TROOP  MOVEMENTS  [ART  2.3] 

4  mm- *;* ^ ^  :  :  I  ►  I 


Name: 

SLAD  Demo  MCS  CO  CAB 2  -  Conduct  an  Attack 

Code: 

US  Army-2004-0084 

Combatant  Command: 

DYNAMICS  RESEARCH  CORPORATION  Modify... 

Last  Modified: 

2004-09-20  1  2:42:31 .0 

Created: 

2004-04-22  10:32:54.0 

*  Subordinate  Command: 

Modify... 

*  OPLAN: 

Modify... 

Published: 

U 

Level: 

<•  Primary  Mission  (M) 

*  Description: _ [zoom... 

Attack  north  on  Axis  Maple  to  occupy  OBJ  Apple.  Establish  attack  by  fire  positions  on  OBJ  Apple.  On  order, 
engage  enemy  forces  already  in  or  entering  EA  Dunk  IOT  prevent  them  from  moving  north  to  support  rebel 
leadership  party  vicinity  ofWest  Point  orto  support  enemy  main  body  defending  in  and  around  Louisville.  End 
State  is  MCS  A  occupying  ABF  positions  on  OBJ  Apple  with  at  least  80%  combat  power  priorto  enemy 
movement  north  from  Knox. 

•  Screen  shot  of  results  of  Mission  to  Task 
decomposition  using  JTIMS  automated 
KA  tool. 

•  Used  to  document  break  down  of  MCS  A 
mission  into  component  tasks. 


[jTIMS  v2.0c  Build  129  (May  7,  2004) 


Java  Applet  Window 


•  Vignette  mission  thread  is  assembled 
from  the  component  tasks. 


Help 

bbray_1 


~L¥T~ 


Operational  Template  from  UJTL,  CJCSM  3500.04D,  dtd.  1 
August  2005 

AIR  ASSAULT 


Description: 

A  forcible  entry  operation  conducted  by  the  US  Army's  air  assault  units  to  seize 
a  lodgment  (normally  an  airfield)  and  defend  it  while  the  JFC  executes  a  rapid 
build-up  of  combat  power. 

Search  Documents:  CJCSM  3500. 04B?  CJCSM  3500. 02B?  JP  1-02,  CJCSM 
3500.04A,  CJCSM  3500.05 

Tasks: 


TASK 

DESCRIPTION 

ST  1.3.3 

Synchronize  Forcible  Entry  in  Theater 

ST  1.3.4 

Integrate  Direct  Action  in  Theater 

ST  5.4.2 

Synchronize  Joint  Operations  and  Subordinate  Campaign  Plans 

ST  6.2.1 

Coordinate  the  Preparation  of  Strategically  Significant  Defenses 

ST  7.1.3 

Tailor  Joint  Forces  for  Deployment 

OP  1.2. 4. 3 

Conduct  Forcible  Entry:  Airborne,  Amphibious,  and  Air  Assault 

OP  1.2. 4. 4 

Reinforce  and  Expand  Lodgment 

OP  1.2.5 

Conduct  Offensive  Operations  in  the  Joint  Operations  Area 

OP  1.2.6 

Conduct  Defensive  Operations  in  the  Joint  Operations  Area 

OP  1.3.1 

Overcome  Operationally  Significant  Barriers,  Obstacles,  and  Mines 

OP  1.5.1 

Control  of  Operationally  Significant  Land  Area  in  the  Joint  Operations  Area 

OP  2.4.1 

Evaluate,  Integrate.  Analyze ,  and  Interpret  Operational  Information 

OP  3.1.7 

Employ  Fire  Support  Coordination  Measures 

OP  3.2.7 

Synchronize  Operational  Firepower 

OP  5.3.1 

Conduct  Operational  Mission  Analysis 

OP  5.3.3 

Determine  Operational  End  State 

OP  5.4.4 

Synchronize  and  Integrate  Operations 

OP  5.5.1 

Develop  a  Joint  Force  Command  and  Control  Structure 

OP  6.1.4 

Counter  Enemy  Air  Attack  (Defensive  Counter  air  j  DC  A}]  in  the  Joint  Operations 
Area 

OP  6.5.5 

Integrate  Host -Nation  Security  Forces  and  Means 

(Operations  Template  Development) 

Systems  Approach  Using  JTS 


Provide 

IDE 

OpnISpt 

Vulner¬ 

abilities 

OP  4 

OP2aa3 

FiJIy 

Partially  1 

Not 

Trained 

Trained  | 

_ 1  Trained 

^  Not 
Evaluated 


Assess 

Reattk 

Reqrmt 

OP3.1.&3 
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Doctrinal  Sources 


FM  3 -SO 

TACTICS 

f 

Sr 

julyzoqi 

HEADQUARTERS 
□B*ARTMENT  QF  THE  ARMY 

Online  Joint  and  Service  digital  libraries  provide 
ready  reference  to  doctrine  and  TTPs  for 
identifying  supporting  tasks 


•  Develop,  select  and  wargame  a  Course  of  Action  COA  to  accomplish  the  mission 

•  Conducted  using: 

•  Maps  of  the  Scenario  area  of  operations 

•  COA  development  and  wargaming  steps  from  Military  Decision  Making  Process  (MDMP) 

•  Laptop  computer  with  In-Focus  projector  to  project  COA  sketch  for  reference  in  conference 
room/work  area 

•  MS  Project  application  software* 

•  Prior  to  wargaming,  one  DRC  SME  set  up  the  Mission  file  for  the  base  COA  in  MS  Project  by: 

•  Setting  initial  parameters  in  terms  of  duration  of  the  vignette  and  time  increments  to  be  used. 

•  Importing  tasks  from  the  step  3  mission  task  report  generated  by  JTIMS  into  the  tasks  portion 
of  the  Mission  file 
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Develop  a  COA 


Input 

•  Mission  received  from  higher  HQ’s  or 
deduced  by  the  commander  and  staff 


•  Higher  HQ  order/pJan/IPB 

•  Staff  estimates 

•  Facts  &  assumptions 


•  Restated  mission 

•  Cdr’s  intent  and  planning  guidance 

•  CCIR 

•  Staff  estimates  and  products 

•  Enemy  COAs 

•  Refined  Cdr’s  planning  guidance 

•  Enemy  COAs 

•  Friendly  COAs 

•  Staff/BOS  COAs 

•  War-game  results 

•  Established  criteria  for  comparison 


•  Decision  matrix 


•  Approved  COA 


Output 


Receipt  of  Mission 

Cdr’s  initial  guidance 
k  •  WARNO  to  staff 

- - ■!:  WARNO  1  .  initial  _ _ _ 

Mission  Analysis 

''  Restated  mission 

t  Crd’s  intent 

'I  Cdr’s  planning  guidance 

Initial  CCIR 

’’  Updated  staff  estimates/products 

•  Preliminary  movement 

•  Initial  ISR  Plan 

COA  Development 

•  Updated  staff  estimates 

•  COA  statements  and 
sketches 

>  Refine  Crd's  planning  guidance 

I  WARNO  | 

COA  Analysis 
(War-Game) 

•  War-game  results 

•  Decision  support  templates 

•  Task  Organization 

•  Mission  to  subordinate  units 

COA  Comparison 

•  Decision  matnx 

"i  Approve  COA 

>l  Refine  Crd’s  intent  and  guidance 
^  Refine  CCIR 

COA  Approval 

WARNO  I 

Orders  Production 

•  OPLAWOPROD 

Note  1 :  A  star  depicts  commander 
activities  and  or  decisions. 

Note  2:  Rehearsals  and  back-briefs 
occur  during  preparation  and  ensure  an 
orderly  transition  between  planning  and 
execution. 


Preparation 


Execution 


_ 

Intuit  \ 

*  Restated  mission  \ 

*  Commanders  intent  \ 

*  Commanders  planning  \ 

guidance  \ 

*  Staff  estimates  &  / 

products  / 

*  Enemy  COAs  {event  / 

templates)  / 

Process 

*  Analyze  relative 
combat  power 

*  Generate  options 

*  Array  initial  forces 

*  Develop  the  initial 
concept  of  operations 

*  Assign  headquarters 

*  Develop  COA 
statements  and  sketches 

Output  \ 

*  Updated  staff  estimates  \ 

*  COA  statements  and  \ 

sketches  \ 

*  Course  of  action  briefing  / 

*  Refined  cdr’s  guidance  / 

Figure  3-8.  COA  Development 


Figure  3-1.  The  Military  Decision  Making  Process 


Will  Convert  COA  into  Mission  Thread  Structure... 


Sample  COA  Sketch  and  Statement 


MISSION: 

Attack  north  on  AXIS  Maple  and  seize 
OBJ  APPLE  NLT  0600  hrs.  Establish 
attack  by  fire  positions  on  OBJ 
APPLE  and  engage  enemy  forces 
already  in  or  entering  EA  DUNK  IOT 
block  enemy  forces  from  moving 
north  to  support  rebel  leadership  vie 
Westpoint  or  support  enemy  forces 
defending  in  and  around  Louisville. 


ENDSTATE: 

Enemy  forces  vicinity  of  Knox  remain 
south  of  EA  DUNK  until  friendly 
operations  vicinity  of  Westpoint  are 
completed. 


© 


-  Targetd  AreaoTlitrest 


I  I  -  MCS  Eigageme  it  Area 
NAI  -  Nan  ed  Area  of  lit  rest 
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Fully  Developed  COA  Statement 


MISSION:  At  170400Z  Ma-ch  03.,  52 d  ID  (Mech:.  atlacks  to  cefeat  elements  of  the 
12th  DTG  in  AO  LION  to  protect  tne  northern  flank  of  tne  55th  AD,  the  21st  (UBj 
Corps  main  effort. 

INTENT :  The  purpose  of  this  attack  is  to  prevert  repostion  ng  of  12:h  DTG  forces  to 
Ihe  south  ard  in:erering  witn  21st  (US}  Corps  decis  ve  operation  (tne  55lh  AD's 
seizure  of  OBJ  STRIKE).  Key  tasks  are 

4  Destroy  73d  Brigace  Taclical  Group  (BTG)  south  of  the  METRO  CITY- 
CENTRAL  TOWN-RIVER  TOWN  Line  to  prevent  tneir  repositioning  south 
into  55th  AD  s  AO. 

4  Seize  OBJ  SLAM  by  1B133DZ  MAR  C-3  to  seoure  tne  norherr  flank  of  the 
55lh  AD. 

4  Defeal  the  12th  DTG  s  reserve  (23d  E T 3. i  vicinity  EAS“  TOWN  1o  prevent 
tnem  from  interfering  with  the  seizure  of  OBJ  SLAM. 

At  enc  s:ate,  the  corps'  right  flank  is  secure  with  two  brigades  cunsolidaled  in 
defense  oositcns  vicinity  OBJ  SLAM,  “he  division  is  prepared  tc  cor  duct  follow-on 
offensive  operations  to  cefeat  enemy  to  PL  RED. 

DECISIVE  OPERATION:  Armor  Bde  £1  passes  through  tne  southern  Mech  Ede  #  1 
east  of  PL  AMBER  anc  attacks  tc  seise  the  key  terrain  vicinity  of  OBJ  S_A.M  in 
order  1o  p -elect  5ctn  AD's  northern  flank. 

SHAPING  OPERATIONS:  Mech  Bde  #1  in  the  soutn  the  initial  main  effort, 
conducts  a  penetration  to  deslroy  enemy  fcrce  vicinity  PL  AMBER  to  create  enough 
maneuver  space  for  Armor  Bee  £1  to  oass  to  the  East  without  interenerce  from  the 
73d  BTG  in  order  to  se  ze  key  terrain  vicinity  uf  OBJ  SLAM  and  prated  Ihe  northern 
flank  of  the  55lh  AC  Armor  Bee  =1  becomes  the  main  effort  after  conducting 
forwa-c  passage  of  ires  with  Mech  Bde#  '  and  Ihen  accepts  battie-hanccve-  along 
PL  GREEN.  Mechanised  Bde  #1  Ihen  Follows  and  supports  Armor  Bde  #1  and  the 
division  reserve  oy  attacking  east  to  clear  rent  a  n  ng  elements  of  the  73d  from  PL 
Amber  to  FL  Green  in  order  tc  protect  the  rear  of  bath  units. 

Tne  division  reserve,  an  armor-heavy  task  fc-ce.  initially  fa  lows  Mechanized  Bde#1 
with  the  fallowing  priorities  of  commitment:  1).  Contain  enemy  forces  capable  of 
threatening  Armor  Bde  =  Ys  passage  through  Mechanised  Ede  £  1  allowing  cattle- 
handover  to  scour  along  PL  Green.  2).  If  not  committed  west  of  PL  Greer  follows 
Armor  Bde  =  1  and  blacks  enemy  force  capable  of  threatening  this  crigades 
movement  east  er  ah  ling  it  to  seize  the  key  terrain  vicinity  of  DBJ  GLAM  ard  p -elect 
the  north  flank  or  the  5ctn  AD. 

Ir  tne  rc-th,  Mech  Ede  #2  attacks  east  to  fix  the  “2c  B“G  denying  it  the  ability  to 
interfere  witn  the  civision's  decisive  operations  in  the  south.  The  division  cavalry 
squadron  conducts  a  moving  flank  screen  alorg  the  division's  nerthe-r  bcundary  to 
provide  ea-ly  warning  o-'  enemy  forces  attacking  scuth  irtc  Ihe  northern  flank  of 
Mech  Bde  3  2. 


COA  *M  (continued  f 

Once  Mech  Bde  #1  crosses  PL  AMBER  (LD),  the  d  vis  on  attack  he  icopte-  battal  on 
(AhB)  attacks  alorg  AIR  AXIS  SIDNEY  lo  destroy  the  enemy  tank  oatta  ion  vicinity 
WEST  TOWN  to  protect  then  nortnem  flank  of  Mech  Bee  #1  and  allowing  il  to  pass 
Armor  Bde  #1  east.  Once  Armor  Bde  #1  accepts  batt  e-handover  along  PL  GREEN 
the  AHG  attacks  along  AIR  A* IS  GIZMO  to  defeat  the  23d  BTG  south  ard  east  of 
the  NORTHERN  MOUNTAINS  to  a  low  Armor  Bde.  #1  to  seize  tne  key  terrain 
vicinity  of  0J6  SLAM  anc  crctec:  the  norherr  flank,  of  the  55tn  AD. 

Division  fires  will:  1}.  Conduct  SE.AD  along  AIR  AXES  SIDNEY  and  GIZMG  to 
allow  the  Al-E  tc  destroy  the  enemy  tank  battalion  vicinity  WEST  TOWN  and  tc 
cefeat  the  23d  BTG  south  and  east  of  Ihe  NORTHERN  MOUNTAINS,  respectively; 
2).  Conduct  counter  f  re  tc  neutralize  two  baltaliors  of  the  "2th  DTG's  Inieg-ated 
F  res  Command  (IFC}  tc  prevent  it  from  massing  fires  against  ihe  southern  two 
brigaces;  3j.  Provide  superessive  fires  against  73d  BTG  defenses  along  PL  AMEER 
to  enable  Mech  #1's  peret-aten. 

Division  ISR  operations  focus  cn  1).  Identifying  the  location  and  disposition  c' the 
73d  BTG  battle  zone  tc  determ  ne  optimal  poinl  of  penetration  for  MECH  Bde  #  1 
along  P_  AMBER:  2).  Location  and  disposition  of  ihe  12th  DTGs  IFC  assets  to 
assist  counter  fire  efforts;  3).  Location  and  intentions  cf  the  enemy  tank  battalion 
and  ADA  assets  vicinity  WEST  TOWN,  and  location  and  intention  of  the  23d  BTG, 
the  enemy's  reserve,  vicinity  the  NOR“HERN  MQUTAIINS,  to  assist  Ihe  AHB 
attacks. 

SUSTAINING  OPERATION.  The  division  support  area  will  establish  vie  n  ty  METRO 
CI“Y  with  MSRs  SEAN  and  NICK  as  the  primary  rcutes  used  tc  sustain  combat 
power  during  tine  attack.  A  mechanized  company  team  is  tine  division  TCF  with 
priority  of  responding  to  any  LEVEL  III  treats  tc  division  class  III  supply  point. 

The  deception  objective  is:  commander  of  tne  12th  DTG  commits  his  reserve,  the 
23d  BTG,  at  H+10  to  block  cenet-at  cn  of  US  fo-ces  in  the  north  cf  AO  LI3N  in  c-ce- 
to  p-otect  tne  24th  DTG,  tine  1st  Field  G -cue's  main  effort.  The  deception  slory  is 
that  tine  civision  s  decisive  c aeration  is  in  the  north,  with  tne  following  indicators:  the 
initial  positioning  of  an  armor-heavy  brigade  in  the  north e-n  portion  cf  Ihe  rea-  are  in 
AA  MIKE,  simultaneous  attacks  of  two  crigades  abreast  in  ihe  north  and  south,  the 
division  cavalry  squacron  operating  on  the  rc-th  flank  c' tne  division  AO,  and  early 
commitment  of  the  division  s  AHB  cest-oy  an  enemy  tank  battalion  n  ihe  north. 

Taclical  risk  is  assumed  by  earty  commitment  cf  tine  division's  Al-E,  potentially 
leaving  it  without  sufficient  combat  power  to  defeat  the  23d  BTG.  the  enemy's 
reserve. 


Figure  3-12.  Course  cf  Action  Statement  (COA  £1) 


Wargame  a  CO  A 


Input 

•  Mission  received  from  higher  HQ’s  or 
deduced  by  the  commander  and  staff 


•  Higher  HQ  order/pJan/IPB 

•  Staff  estimates 

•  Facts  &  assumptions 


•  Restated  mission 

•  Cdr’s  intent  and  planning  guidance 

•  CCIR 

•  Staff  estimates  and  products 

•  Enemy  COAs 

•  Refined  Cdr’s  planning  guidance 

•  Enemy  COAs 

•  Friendly  COAs 

•  Staff/BOS  COAs 

•  War-game  results 

•  Established  criteria  for  comparison 


•  Decision  matrix 


•  Approved  COA 


Output 


Receipt  of  Mission 

^  Cdr’s  initial  guidance 
k  •  WARNO  to  staff 

- - ■!:  WARNO  1  .,n^iDR _ _ 

Mission  Analysis 

Restated  mission 
>  Crd’s  intent 
'I  Cdr’s  planning  guidance 

Initial  CCIR 

’’  Updated  staff  estimates/products 

•  Preliminary  movement 

•  Initial  ISR  Plan 

COA  Development 

•  Updated  staff  estimates 

•  COA  statements  and 
sketches 

't  Refine  Crd's  planning  guidance 

I  WARNO  | 

COA  Analysis 
(War-Game) 

•  War-game  results 

•  Decision  support  templates 

•  Task  Organization 

•  Mission  to  subordinate  units 

COA  Comparison 

•  Decision  matnx 

^  Approve  COA 

'I  Refine  Crd’s  intent  and  guidance 
>  Refine  CCIR 

COA  Approval 

WARNO  I 

Orders  Production 

•  OPLAWOPROD 

Note  1 :  A  star  depicts  commander 
activities  and  or  decisions. 

Note  2:  Rehearsals  and  back-briefs 
occur  during  preparation  and  ensure  an 
orderly  transition  between  planning  and 
execution. 


Preparation 


Execution 


Input  \ 

*  Staff  estimates  \ 

*  IPB  (enemy  COAs)  \ 

*  COA  statement  and  \ 

sketch  / 

*  Supporting  staff  / 

functional  COAs  / 

Process 

•  Gather  the  tools 

•  List  all  friendly  forces 

•  List  assumptions 

•  List  known  critical 
events  and  decision  points 

•  Determine  evaluation 
criteria 

•  Select  the  wargame 
method 

•  Select  a  method  to 
record  and  display  results 

•  War-game  the  battle 
and  assess  the  results 

Ol(!l)lif  \ 

*  Wargame  results  to  \ 

include—  \ 

•  Concept  of  operations  \ 

*  Synchronization  matrix  \ 

*  Operations  overlay  J 

•  Decision  support  / 

template  j 

•  Task  organization  / 

*  Updated  CCIR  j 

Figure  3-12.  Course  of  Action  Analysis  (War  Game} 


Figure  3-1.  The  Military  Decision  Making  Process 
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Associate  Tasks 
With  Conditions 
&  Measures/ 
Standards 


Step  4 


Continued 


•  During  the  wargaming  process,  record  the  results  of  the  wargaming  by: 

•  Ordering  mission  tasks  in  the  wargamed  sequence 

•  Annotating  the  task  purpose,  conditions,  Measures  of  Effectiveness  (MOE),  and 
Measures  of  Performance  (MOP)  for  each  task  to  be  observed  and  evaluated,  based 
on  wargaming  team  discussion  and  consensus. 

•  Upon  completion  of  initial  wargaming  process,  review,  critique  and  modify  above 
results  as  required.  Focus  should  be  on  the  following  questions: 


•  Did  we  select  the  most  appropriate  task  to  describe  the  activity? 

•  Is  the  purpose  of  the  task  clearly  stated  and  does  it  make  good  tactical 
sense? 

•  Do  the  MOEs  support  ability  to  determine  if  the  purpose  is  achieved? 

•  Do  the  MOPs  support  and  flow  logically  from  the  MOEs  and  are  they 
objectively  measureable? 
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Wargaming  generates  detailed  and 
valuable  information  -  Who  knew? 


3-187.  An  effective  wargame 
results  in  the  commander  and 
staff — 

•  Refining  or  modifying  each  COA, 
including  identifying  branches  and 

sequels  that  become  on-order  or 
be-prepared  missions. 

•  Refining  the  locations  and  times 
of  decisive  points. 

•  Identifying  key  or  decisive  terrain 
and  determining  how  to  use  it. 

•  Refining  the  enemy  event 
template  and  matrix. 


■M  5-0 

Eefiomf  che  cask  organization.  including  forces  retained  in  general 
support. 

Identify  tag  tasks  che  unit  reraias  and  “asks  as- signed  to  auk  ordinates. 

Allocating  assets  to  subordinate  commanders  to  accomplish  their 

missions. 

Developing  decision  points. 

Developing  a  synchronisation  mains. 

Developing  -decision  support  template. 

Estimating  the  duratioa  of  the  entire  operation  and  each  critical  event. 

Projecting  the  percentage  of  enemy  forces  defeated  in  each  critical 
event,  and  overall. 

Identifying  likely  times  and  areas  for  enemy  use  of  WMD  and  friendly 

XBC  defense  requirements. 

Identifying  the  potential  times  or  locations  for  committing  the  reserve. 

Identifying  the  meet  -dangerous  enemy--  COA. 

Identifying  locations  for  the  commander,  command  posts,  and 

INFO  SYS  nodes. 

Identifying  -critical  events. 

Identifying  requirements  for  BOS  suppert. 

Determining  requirements  for  military-  deception  and  surprise. 

Refining  C!2  requirements,  including  control  measures  and  updated 
operational  graphics. 

Refining  GCIE  and  EEL — including  the  LTIOV — and  incorporating  them 
into  the  ISE  plan  and  Information  Management  plans. 

Developing  che  ISR  plan  and  graphics. 

Developing  IG  objectives  and  tasks  (see  FM  3-13). 

Developing  fire  support,  engineer,  air  defense.  IO.  and  CSS  plans  and 
graphic  central  measures. 

Identifying  the  effects  of  friendly  and  enemy  action  on  the  civilian 
population  and  infraacructure.  and  how  these  will  affecc  milLcary 

operations. 

Identifying  or  confirming  the  locations  of  NAIsf  TAIs,  decision  points, 
and  ER  needed  to  support  them. 

Determining  the  timing  for  concentrating  fortes  and.  acarting  the 
attack  or  counterattack. 

Determining  movement  times  and  tables  for  critical  assets,  including 

INFO  SYS  nodes. 

Identifying,  analyzing,  and  evaluating  strengths  and  weaknesses  of 
each  COA. 

Integrating  targeting  into  the  operation,  to  include  identifying  or 
confirming  high-payoff  targets  and  estahliahing  attack  guidance. 

Identifying  hazards,  assessing  their  risk,  developing  controls  for  them, 
and  determining  residual  risk. 

Wargaming  Notes 


CRITICAL  EVENT:  Aviation  Deep  Attack 

Sequence = 

Action 

AVN  BDE  attacks  to  destroy  enemy  reserve  -on  OBJ  DEBRA 

React  cn 

Anti -air  ambush  in  route 

Counteraction 

J/SEAD  change  air  axis 

Assets 

1-7S  FA  (MLRS),  1-222  (AHB),  Fixed-Wing 

Time- 

H-2  to  H  +  1 

Decision  Point 

DP  3b  and  3c 

CCIFL 

Location  of  enemy  armor  reserve  south  of  PL  Kansas 

Control 

Air  Axis  Falcon  and  Eagle 

Remarks 

Figure-  3-17,  Sketch  Note-  Work  Sheet 

Detail  for  one  event 
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Task  and  Purpose 


Actions  by  Friendly  Forces 


A,1tao<-by-Fire 

B'each 

Bypass 

Clear 

Consolidate  and 

Reorganize 

Control 

Disengage 

Follow  and  Assume 

-  ollcv^  and  Support 

L  nkup 

Occupy 

deduce 

Retain 

Secure 

Seize 

Su  pport  by  F  re 


Effects  on  Enemy  Forces 


B  ock 

Cana  ze 

Contain 

Defeat 

Destroy 

Disrupt 

rix 

nterdicl 
so  late 
Neutralize 
Penet'ale 
Turn 


Figure  3-7.  Mission  Task 


Conditions 


•  Conditions  are  variables  of  the  operational 
environment  or  situation  that  may  affect 
performance. 


•  Environment  is  the  immediate  situation  or 
circumstance  in  which  tasks  must  be  performed. 


Conditions  impact  the  ability  to  perform  a  task  (e.g., 
security  of  lines  of  communication  impact  the  ability 
to  sustain  joint  task  force  operations). 


54 


Measures 


Measures  distinguish  among  varying  levels  of 
task  performance.  More  than  one  measure  may 
be  specified  for  any  single  task. 

Task: 

OP  2.2.1  Collect  Information  on  Operational  Situation 

Measures: 


SCALE 

MEASURE  | 

Time 

To  retask  collection  asset 

Time 

Since  most  current  intel.  info,  was  collected 

Percent 

Of  collection  requirements  filled 

Percent 

Of  collection  reqmts  filled  by  multiple  sources 

Percent 

Of  targets  accurately  located 

Percent 

Of  targets  accurately  identified _ 
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Standards 


Standards  express  the  degree  to  which  (how  well)  a 
military  organization  or  force  must  perform  a  task 
under  a  specified  set  of  conditions. 

A  criterion  defines  acceptable  levels  of  performance 
for  a  measure  and  is  often  expressed  as  a  minimum 
acceptable  level  of  performance. 

Standard: 


Criterion 

Scale 

Measure 

90 

Percent 

of  collection  requirements  filled 

4 

Hours 

since  most  current  intelligence 

information  was  collected 

85 

Percent 

of  targets  accurately  located 

i 
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MS  Project  Mission  thread  w/  notes  page 


Microsoft  Project  -  SLAD-TOEL  Mission  Thread(20  Jul) 


J 

'■  Fie  Edit  View  Insert  Format  Tools  Project  Window  Help 
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|  Hour  1 

(Hour  2 

Hour  3 

|  Hour  4 

|  Hour  5 

Hour  6 

Hour  7 

Hour  8  ^ 

* 

/ICS  A  Collective  Tasks 

Task  Name 

B  CAB  2 

"eT  NLOS-Mortar 


ART  3  3.1  1  Conduct  Surface  to  Surface  Attack 


MTP  06-5-A008  Conduct  Fire  Missions 


B  Security  Force  (1  MCS,  1  ARV) 

LSI  A1.5.2  Occupy  an  Attack/Assaul 
LSI  A1.2  Conduct  Tactical  Maneuver 
LSI  A1.2.3.3  Exploit  Terrain  to  Exped 
LSI  A1.2.4.7.3  Negotiate  a  Tactical  A 
MTP  17-5-001 1.1 7-KCRW  Establish  « 
LSI  A4.2.1  Conduct  Tactical  Reconai: 
LSI  A4.2.2  Conduct  Surveillance  ART 
~i  IpIt/MCS 

LSI  A1.5.2  Occupy  an  Attack/Assaull 
LSI  A1.2  Conduct  Tactical  Maneuver 
LSI  A1 .2.3.3  Exploit  Terrain  to  Exped 
LSI  A1.2.1.1  Employ  Travelling  Mover 


Task  Information 


General 


B  MCS  CO  A 

MCS  A  LSI  A1.5.2  Occupy  an  Attack/Assault  Position  ART  2.5.2 
MCS  A  LSI  A1  2  Conduct  Tactical  Maneuver  ART  2.2 
MCS  A  MTP  17-5-00 11.1 7-KCRW  Establish  and  Maintain  Commi 


MCS  A  LSI  A4.2.1  Conduct  Tactical  Reconaissance  ART  1  3.3 


MCS  A  LSI  A4.2.2  Conduct  Surveillance  ART  1  3.4 
MCS  A  ART  7.4.1  CONDUCT  MDMP 
COA  #1  (Transfer  UAV  Control  to  1st  and 
COA  #2  (Transfer  Control  of  UAVs  to  FTT 
COA  #3  (Request  UAV  Support  from  the 
B  C2V 

LSI  A1  5  2  Occupy  an  Attack/Assaul 
LSI  A1  2  Conduct  Tactical  Maneuver 
LSI  A1  2  3.3  Exploit  Terrain  to  Exped 
LSI  A1.2.1.1  Employ  Travelling  Mover 
LSI  A1.2.4.7.3  Negotiate  a  Tactical  A 
MTP  07-1-1  COP.  07-C332  Establish  tl 
ART  7.2.5  Disseminate  Common  Opr 
LSI  A2.3  1  Collect  Relevant  Informatic 
MTP  07-1-WT06.07-C332  Conduct  Bi 
MTP  07-1-3000. 07-C332  Employ  Fire 
MTP  17-5-001 1.1 7-KCRW  Establish  2 
LSI  A1. 6. 2. 1.1. 4. 3  Report  Enemy  Info 


|  Predecessors  | 

Resources 

Advanced 

Notes  | 

Name: 


Notes: 


|  MCS  A  LSI  A4.2.1  Conduct  T actical  Rec 

Duration: 

[405m 

3 

Custom  Fields 


IT  Estimated 


urpose:  To  find  and  report  enemy  presense  and  activity  along  Axis  Maple  and  on/a  round 
Obj  Apple  and  report  information  about  enemy  activities. 

Conditions:  MCS  A  assets  have  been  assigned  areas  of  responsibility  to  conduct 
reconnaissance.  Both  digital  and  voice  communications  are  established.  Net  procedures 
are  in  place.  MCS  A  has  an  identified  axis  of  advance.  MCS  main  body  is  in  AP 
Muldraugh.  There  is  50%  illumination  at  night  and  UAV  flight  patterns  have  been  established. 
The  enemy  is  occupying  defensive  positions  south  of  West  Point  and  there  are  likely  enemy 
forces  on  the  objective.  They  are  light  dismounted  infantry.  The  terrain  is  woodland  and 
rolling  hills. 

Standards: 

MOE  Scale  Measure 

01  Yes/No  MCS  A  reconnaissance  assets  sensed  and  reported  all  enemy  activity  along 
Axis  Maple  and  on  Objective  Apple. 

MOP  Scale  Measure 


Help 


OK 


Cancel 


i . i 

;king 

. i 

J* 


Ready 
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Sample  Wargaming  Output  -  Synch 

Matrix 


to 

1 

I 

H  —  hour 

I 

+ 

TO 

Enemy  Action 

e  my  M  c  nitors 
Movem  e  nrs 

□e-'ends  frnm  Seouriry 

21  c  ne 

Comm  ts  Reserve 

Launch  Beep  Arrack 
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E-ion  Fo-imts 

M 

1st  Bde 

Move  c-n  Rcure  Paula 

Cross  LEi 

Seize  cn  OBJ  N  ck 

A 

N 

Move  on  Route  Mike 

Cross  LO 

E 

U 

3rd  Bde 

Move  c-n  Route  Sean 

FPCL  wilh  1st  BDE 

V 

E 

Avn  Bde 

Deep  .Attack  on  CB_  Rose 
:tC|  H  -  1 

R 

Div  C  av 

Soreen  Ncrth  Flank 

Wea  pons  H  o  Id 

Air  D  E*f e  n  s  e 

Prep  Fires  Iniriarec  al  H  -  £ 

Fire-  Support 

1  nff-orm  at  io  n 
□  perat  ions 

Su  rre  nder  B  roa  d  casts 

E  nemy  -C  2,  Ja  mr  ed 

M/C 

M/B 

Route  Maintained 

CBS 

MSP  T-amc-a  Closed 

South  hour  d 

C2 

TAC  CP  witn  Lead  Bde 

MOTE: 

The-  fin*t  co-lumi 

n  Be-  re preseii tat Ive  only  and  can  be 

i  mDdiriHd  to  lit  unit  neads. 

Figure  3-16.  Sample  Synchronization  Matrix  Format 


Recording  Wargame  results  in  Mission 

Thread 


•  Determine  the  type  and  level  of  functions  and  capabilities  required  to  enable  tasks 
to  satisfy  MOPs  under  the  conditions  identified 

•  Primarily  a  cognitive  process  performed  by  SMEs  applying  logic,  military 
judgment  and  experience 

•  Function  and  Capability  Descriptions  vary,  particularly  when  describing 
functions  and  capabilities  of  formations/organizations  versus 
components/platforms  (including  people) 

•  Requirements  documents  for  Systems  and  Systems  of  Systems  provide 
detailed  description  of  functions  and  capabilities  to  be  delivered  (and  tested) 

•  Bottom-up  execution  and  assessment  portion  of  MMF  requires  ability  to 
instantaneously  compare  “required”  to  “available”  capability  during  execution  of 
mission  threads 


Sample  Cognitive  Process  for 
Determining  Required  Capabilities 


Elements  of 
Combat  Power 

Enemy  strengths/  Friendly  strengths/  1 

Advantage 

weaknesses 

weaknesses 

Friendly 

Enemy 

MANEUVER 

Strength:  Infantry  with 
numerous  ant-:ank 
weapons. 

Weakness:  Poorly 
ma  ntainec  equipment. 
Lack  □ f  rrob  Iby 
between  battle 
positions 

Strength:  3X  M1A2 
equ  p  combined  arms 
task  fences. 

X 

FIREPOWER 

Weakness:  Limbed  to 
rrmrtar  fires. 

Strength:  Air 
supremacy,  unc-p  cosec 
CAS,  rocket  and 
cannon  fires. 

X 

PROTECTION 

Strength:  Fully 
construcred  defensive 
position  with  ove-head 
cover. 

Strength:  N  ahl  vis  on 
capability ;  weapons 
s:andoif. 

Weakness:  Soft  skin 
vehicles  anc 
cismoumed  nfantry. 

X 

LEADERSHIP 

Stnenotn  Elbe  unit  very 
disciplined. 

Weakness:  Lack  of 
initiative  by 
subc-cinates  with  □  ul 
orders  from  higher 
command. 

Strenatn:  Combat 
tested  unit.  Aggress  ve 
and  offensive  oriented 

command  climate. 

X 

INFORM  AT  1 0  N 

Etnenntn:  Full  cackina 
of  coa  population  and 
regional  p-ess 
Weakness:  C2  very 
acceptable  to  jamming 
and  menception. 

Ctrenatn:  Secure  and 
reliable  C2  systems. 
Weakness:  Seen  as 

X 

invaders  and  occupiers 
by  op  cosing  fc-ce  and 
local  population 

Figure-  3-9.  Sample  Elements  of  Combat  Power  Analysis 


Describing  Levels  of  Capability  -  Degraded 

Capability  Elements 


C2V 


Mobility 

m1  Reduced  maximum  speed 
m2  Reduced  maneuverability 
at?3  Stop  after  f  min  (leaks) 

A7?4  Total  immobilization 


Surveillance.  & 
Reconnaissance 

Z1  Lost  primary  sensor 
Z2  Lost  secondary  sensor 
Z3  Lost  tertiary  sensor 
Z4  Lost  vision  blocks 


NLOS 

(6) 


Firepower 

Lost  ability  to  fire  buttoned-up 
f2  Degraded  delivery  accuracy:  main 
f3  Degraded  initial  rate  of  fire:  main 

f4  Degraded  subsequent  rate  of  fire: 
main 

f5  Total  loss  of  firepower:  main 


Class-ll  UAV 


(3) 


Target  Acquisition 

a1  Lost  daylight  sights 
a2  Lost  night  sights 


Communication 

X1  Lost  external  data 
X2  Lost  external  voice 
X3  Lost  internal  comms 
X4  Lost  LAN 
X5  Lost  all  comms 


Survivability 

51  Lost  NBC  protection 

52  Lost  ability  to  deploy 
obscurants 

53  Lost  silent-watch  capability 

54  Lost  APS 

55  Lost  secondary  armament 


Crew 

C1  Commander  incapacitated 
C2  Squad  leader  incapacitated 
C3  Driver  incapacitated 
C4  Operator  1  incpacitated 
C5  Operator  2  incapacitated 
Cq  Gunner  incapacitated 
Cj  Loader  incapacitated 


Other  Mission  Functions 

01  Lost  situational  awareness 


Catastrophic  Loss 

/0|  Lost  every  capability  (fuel  fire, 
annmo  detonation,  ...) 
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O&O  Capabilities  Mapped  to  Materiel  via  ORD  &  PCDs 

(What  materiel  will  be  assessed  to  evaluate  capability  of  interest?) 


Capabilities  provides 

as  V  \ 

supplied  i 

by  ^  Materiel 


O&O  Capability 

ORD  Reference  # 

Procurement  Control  Drawings  (PCD) 

FI 5:  Enable  continuous  situational 
understanding  through  networked  force 

2392 

786-300151 

2314 

786-30093,  Integrated  Computers;  786-30143,  Network  Management 

2365 

786-30093,  Integrated  Computers;  786-30143,  Network  Management;  786-30151 

2661 

786-30018,  Class  III  UAV;  786-30019,  Class  II UAV;  786-30020,  Class  I UAV;  786- 
30028;  786-30146,  Situation  Understanding  (SU)  SSS;  786-30151 

2869 

786-30131,  Tactical  UGS  System;  786-30132,  Urban  UGS  System;  786-30144,  C2  Battle 
Command  &  Mission  Execution  SSS;  786-30145,  C2  Planning  &  Preparation  SSS;  786- 
30151 

FI 6:  Facilitate  automatic  language 
translation 

1040 

786-30093,  Integrated  Computers 

FI 7:  Enable  information  exchange  via 
adaptive,  integrated  communications 

1016 

786-300144,  C2  Battle  Command  &  Mission  Execution  SSS;  786-300145,  C2  Planning  & 
Preparation  SSS;  786-300148;  786-300149,  Warfighter  Machine  Interface  (WMI);  786- 
300151;  WIN-T;  JTRS 

1120 

786-30140,  Ground  Vehicle  Communications;  786-30143,  Network  Management;  786- 
30149,  Warfighter  Machine  Interface  (WMI);  786-30151 

1148 

786-30018,  Class  III  UAV;  786-30019,  Class  II  UAV;  786-30020,  Class  I  UAV;  786- 
30022,  Armed  Robotic  Vehicle  (ARV);  786-30025,  Small  UGV  (SUGV);  768-30093, 
Integrated  Computers;  786-30151 

Comparing  Required  to  Available 
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Test  and  Evaluation  Perspective  -- 

(What  must  be  done  in  a  test  to  enable  evaluation  of  capabilities?) 


O&O  Capabilities 


Tasks 


to  perform 


enabled  Capabilities 
by 


TASK 


Manage  Tactical  Information 


Process  Relevant  Information  to  Create  a  Common 
Operational  Picture 


Display  a  Common  Operational  Picture  (COP) 
Tailored  to  User  Needs 


ART  7.2.4  Store  Relevant  Information 


Disseminate  COP  and  Execution  Information  to 
Higher,  Lower,  Adjacent,  Supported,  and  Supporting 
Organizations 


ART  7.2.6  Communicate  with  Non-English  Speaking 
Forces  and  Agencies 


AUTL  Ref# 


ART  7.2 


ART  7.2.2 


ART  7.2.3 


ART  7.2.5 


c 

o 

03 

c 

E 

CD 
( f. ) 
C/5 
TD 
0) 
_Q 
03 
C 
LU 


CD 

E 

CD 

_C 

O 

C/5 


Red  with  an  X  =  Direct  enabler  of  intent  of  task 
Yellow  with  a  /  =  Enables  performing  task  to  standard 
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Tasks  .^^Capabilities: 
Linking  it  all  Together 


1 

Lines  from  the 

TOEL 

2 

TASKS 

PLATFORM 

51 

0200-1000 

ART  7.2 

*MTP  17-5-001 1.17-KCRW  Establish  and 
Maintain  Communications 

ARV  2 

52 

0412-0417 

ART  7.2 

*LSI  A1 .6.2.1 .1.4.3  Report  Enemy 
Information 

ARV  2 

53 

0200-1000 

ART  7.2 

*MTP  17-5-001 1.17-KCRW  Establish  and 
Maintain  Communications 

ARV  3 

54 

0757-0802 

ART  7.2 

*LSI  A1. 6. 2. 1.1. 4. 3  Report  Enemy 
Information 

ARV  3 

55 

0200-1000 

ART  7.2 

*MTP  07-1-1COP. 07-C332  Establish  the 
Common  Operational  Picture 

C2V 

56 

0200-0205,  0253-0253, 
0308-0313,  0341-0346, 
0437-0442,  0525-0530, 
0633-0638,  0707-0712, 
0300-0305,  0349-0354 

ART  7.2 

*ART  7.2.5  Disseminate  Common  Operational 
Picture  and  Execution  Information 

C2V 

57 

0200-1000 

ART  7.2 

LSI  A2.3.1  Collect  Relevant  Information  ART 
7.2.1 

C2V 

5S 

0200-1000 

ART  7.2 

MTP  07-1 -WTO 6. 07-0332  Conduct  Battle 
Tracking 

C2V 

Com  ms 


59 


60 


61 


62 


63 


64 
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0200-1000 

0255-0300 

0339-0344 

0523-0528 

0706-0711 

0844-0849 

0210-0542 

0250-0255 

030-0335 

0340-0335 

0431-0436 

0543-1000 


H  4  ►  ►l\TI 

n mu  -  k- 


Mobility 


Surveillance.  & 


f 


NLOS 

(6) 


ARV-RISTA 

(3) 


Class-1 1  UAV 
(3) 


m1  Reduced  maximum  speed 
m2  Reduced  maneuverability 
/TJ3  Stop  after  t  min  (leaks) 
mA  Total  immobilization 

rxcoum  idioadi 

Z1  Lost  primary  sensor 

Z2  Lost  secondary  sensor 

Z3  Lost  tertiary  sensor 

Z4  Lost  vision  blocks 

Firepower 

f,  Lost  ability  to  fire  buttoned-up 
f2  Degraded  delivery  accuracy:  main 
f3  Degraded  initial  rate  of  fire:  main 
fA  Degraded  subsequent  rate  of  fire: 
main 

f5  Total  loss  of  firepower:  main 

Communication 

x1  Lost  external  data 
x2  Lost  external  voice 

X3  Lost  internal  comms 

X4  Lost  LAN 

X5  Lost  all  comms 

Target  Acquisition 

a i  Lost  daylight  sights 
a2  Lost  night  sights 

Survivability 

51  Lost  NBC  protection 

52  Lost  ability  to  deploy 
obscurants 

53  Lost  silent-watch  capability 

54  Lost  APS 

55  Lost  secondary  armament 

Crew 

C-i  Commander  incapacitated 
C2  Squad  leader  incapacitated 
C3  Driver  incapacitated 
C4  Operator  1  incpacitated 
C5  Operator  2  incapacitated 
C6  Gunner  incapacitated 
C7  Loader  incapacitated 


Other  Mission  Functions 

0 1  Lost  situational  awareness 


Catastrophic  Loss 

k. |  Lost  every  capability  (fuel  fire, 
ammo  detonation,  ...) 


xQ 

x2  x3  x4  | 

xO 

x2  x3  x4  1 

xO 

x2  x3  x4  I 

xO 

x2 

x3  x4 

xO 

xl 

x2 

x2 

x3 

x4 

x5 

xO 

xl 

x3 

x4 

x5 

xO 

xl 

x2 

x3 

x4 

x5 

xO 

xl 

x2 

x3 

x4 

x5 

xO 

xl 

x2 

x3 

x4 

x5 

xO 

xl 

x2 

x3 

x4 

x5 

xO 

xl 

xO 

xl 

xO 

xl 

xO 

xl 

xO 

xl 

i 

•  Determine  forces  and  associated  MTOE  type  data  that  are  to  be  included  in  the  test. 

•  Test  Objectives  and  nature  of  the  test  (i.e.  DT,  OT,  etc.)  or  test  event  may  drive 
determination. 

•  Use  available  source  documents  (i.e.,  ORD,  O&O,  Systems  Book,  etc.)  to 
understand  organization  and  platform  required  capabilities,  limitations,  intended 
use,  etc.  Incorporate  this  information  into  wargaming  and  determination  of  task 
MOPs. 

•  ARL-SLAD  used  the  documentation  above  and  other  technical  documentation  to 
generate  or  select  substitute  platform  (i.e.  Stryker,  Paladin,  etc.)  tables  of  capability 
categories  and  degraded  capability  elements  to  support  analysis  of  platforms  to  be 
modeled 
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MCS  A  Company  Organization 


Mounted  Combat  System  Company 

Mounted  Combat  System  Company  X  2 


_L_ 

lies 


r 


MS 


MO§  Co  »  Section 


MCS 

*3 

ARV~ 

RSTA 

9  TOE  Per  tunnel 
1  MCS 

i  ifw-rstm 


10  TOEPflnm'inBi 

i  c 

1  MCS 

1  FT  TS-Lf  iCSj. 

3  WW  CL  II  L.C  JnNb- 
3UAVO.NAaririVWiKfa* 


TRADOC  PAM  525-3-90 
Maneuver  O  &  O,  June  2003 


37  TOE  Personnel 


w  MounM combat  syttomt  pcs] 

t  kv 

3  ARV-RSTAs  rs  'Ton  Variant) 

1  FTTB’U  (GE) 

3  UflV  CL  II  UC  Units 

SUAVCLMAwiolMiides 


MCI  Plrtoon 

1  m 


UICS 


MCS 


tt  t 

PL  GNRCRWCHF 
m  ES  Fn 
ISA  13KEA  ISHIu 

ft  t 

VC  DHH  CRWCHF 
Efl  FR  E4 

I  yiiilJl'jHltij  HMu 

1 1 t 

FIG  GNRCHWCHF 
£7  Li  L-l 
1-El  nil  ISKftl  ISKltl 


ARV-R3TA 


MCS  Platoon 

Wffi 


MCS  Platoon 

fm 


MGS 


MCS 


tt  t 

PL  □  HH  CHW  CHF 

■K  Et  CU 

19A13H3Q  13K1H 

tit 

VC  CHHCRWCHF 

St  IH  ss 
1Pl^DieH£D  1-&H10 

t  t  t 

PSG  c  HH  CHVU  CHF 
FT  ES  Ed 
isn«iana]  iskio 


MCS 


+  .  hies 


ft  t 

PL  GNRCRWCHF 
LLt  ES  E4 
t  A  IGHM  1SH1A 

ttt 

VC  DHHCFTlPCHF 

Ffl  E5  E4 
1SKSFH3HM  HMD 

1 1 1 

PEG  GNRCRWCHF 
£7  £5  L-1 

riKJA  ISKttl  1SH1A 


ARV-R0TA 


MCS 


f  t  t 

'.UK  U  HH  LH w 
UJ  ES  CNF 
llfii  11  HH  rJ 
19M10 


Company  HQ  Section 
2  JOB 


ftts-u  re  2] 


1 1 

ISO  5PLr 

CO  rn 


ftttt 

tq  fires  Aoentcs  Vc-  gjvsIg 
H?  FR  F1:  FHFM  Cl'/n 

13*  13FSO  13K20  ES  Ei. 

5+HM  711191 


•  From  T&E  perspective  this  will  normally  be  provided  in  Requirements  Documentation 

•  Can  be  a  challenge  to  pull  together  data  for  current  systems  and  forces 

•  Especially  challenging  to  gather  data  for  degraded  capability  caused  by  changes  in 
systems  and  forces 
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Testing  and  Evaluation  Perspective  -- 

(What  materiel  will  be  assessed  to  evaluate  capability  of  interest?) 


Capabilities  provides 

as  V  \ 

supplied  i 

by  ^  Materiel 


O&O  Capability 

ORD  Reference  # 

Procurement  Control  Drawings  (PCD) 

FI 5:  Enable  continuous  situational 
understanding  through  networked  force 

2392 

786-300151 

2314 

786-30093,  Integrated  Computers;  786-30143,  Network  Management 

2365 

786-30093,  Integrated  Computers;  786-30143,  Network  Management;  786-30151 

2661 

786-30018,  Class  III  UAV;  786-30019,  Class  II UAV;  786-30020,  Class  I UAV;  786- 
30028;  786-30146,  Situation  Understanding  (SU)  SSS;  786-30151 

2869 

786-30131,  Tactical  UGS  System;  786-30132,  Urban  UGS  System;  786-30144,  C2  Battle 
Command  &  Mission  Execution  SSS;  786-30145,  C2  Planning  &  Preparation  SSS;  786- 
30151 

FI 6:  Facilitate  automatic  language 
translation 

1040 

786-30093,  Integrated  Computers 

FI 7:  Enable  information  exchange  via 
adaptive,  integrated  communications 

1016 

786-300144,  C2  Battle  Command  &  Mission  Execution  SSS;  786-300145,  C2  Planning  & 
Preparation  SSS;  786-300148;  786-300149,  Warfighter  Machine  Interface  (WMI);  786- 
300151;  WIN-T;  JTRS 

1120 

786-30140,  Ground  Vehicle  Communications;  786-30143,  Network  Management;  786- 
30149,  Warfighter  Machine  Interface  (WMI);  786-30151 

1148 

786-30018,  Class  III  UAV;  786-30019,  Class  II  UAV;  786-30020,  Class  I  UAV;  786- 
30022,  Armed  Robotic  Vehicle  (ARV);  786-30025,  Small  UGV  (SUGV);  768-30093, 
Integrated  Computers;  786-30151 

Matching  Components  to  Required 

Capabilities 


C58 


A 

B 

c 

D 

E 

F 

G 

\  Capability  Category 

MOBILITY 

FIREPOWER 

COMMUNICATIONS 

SURVIVABILITY 

TARGET 

ACQUISITION 

SURVEILLANCE 

Platform  \ 

A  quality  or  capability  of 
military  forces  which  permits 
them  to  move  from  place  to 
place  while  retaining  the  ability 
to  fulfill  their  primary  mission. 

The  amount  of  fire  which  may 
be  delivered  by  a  position,  unit, 
or  weapon  system.  2.  Ability  to 
deliver  fire. 

The  means  or  method  to  convey 
information  of  any  kind  from  one  person 
or  place  to  another. 

Concept  which  includes  all 
aspects  of  protecting  personnel, 
weapons,  and  supplies  while 
simultaneously  deceiving  the 
enemy.  Survivability  tactics 
include  building  a  good  defense; 
employing  frequent  movement; 
using  concealment,  deception, 
and  camouflage;  and  constructing 
fighting  and  protective  positions 
for  both  individuals  and 

equipment. 

The  detection,  identification, 
and  location  of  a  target  in 
sufficient  detail  to  permit  the 
effective  employment  of 
weapons. 

The  systematic  observation  of  aerospace,  surface,  or  subsurface 
areas,  places,  persons,  or  things,  by  visual,  aural,  electronic, 
photographic,  or  other  means. 

AIRCRAFT 

Rotary  Ving 

X 

X 

X 

X 

X 

Fised  Ving 

X 

X 

X 

X 

UAVs 

X 

X 

X 

X 

X 

X 

Air  Traffic  Control  Sgstems 

X 

X 

X 

Aircraft  Rockets 

X 

X 

X 

AIR  DEFENSE  SYSTEMS 

X 

X 

X 

X 

X 

ANTIARMOR  WEAPONS 

X 

X 

X 

X 

C4I  SYSTEMS 

Digital  Battle  Command 

X 

C4  Support  to  Air  &  Missile  Defense 

X 

X 

X 

C4  Support  to  Network  Operations 

X 

C4  Support  to  Intelligence  Operations 

X 

X 

X 

Sensor  f  Sensor  Sgstems 

X 

X 

X 

X 

Night  Vision  Sensors 

X 

X 

X 

X 

Aircraft  Survivabilitg  Equipment 

X 

Radios  &  Communication  Sgstems 

X 

X 

ENGINEER  &  COUNTERMINE 
EQUIPMENT 

|  Engineer  Equipment _ 

X 

X 

X 

A k kl lA _ r i  irrpnf  Pbl-fnrmc /  Pbt-fnrm  Tn  Al  ITI  Tacke  \  Platform  T at  to  C anahilitA/  / Prinril^/  Malriv _ /  TOT  AIITI  ft  All  PI  ATFORMC, /  Al  ITI  TNPl  ft  All  PI  1  4 _ I _ L_ 


Associate 
Components 
With  Tasks 
Step  9 


•  From  T&E  perspective,  the  O&O  plan  and  follow  on  documentation  such  as 
Operational  Mode  Summary  -  Mission  Statements  will  provide  the  association  for 
specific  mission  sets 

•  This  information  is  useful  for  constructing  mission  threads  and  TOELs  for  use  in  test 
events  and  for  tying  together  discrete  test  events  (i.e.  live  fire  or  developmental  tests) 
being  performed  as  part  of  a  larger,  distributed  test  construct. 
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Matching  Components  (Platforms  and 
Units)  to  Tasks  -  Task  Organize 


M91 


ART  1.3.3  CONDUCT  TACTICAL  RECONNAISSANCE 


Link  to  Equipment  Data  Sheet 


Chapter  1  ART  1.0:  The  lntel]iqence  Battlefield  Operating  System 


SECTION  I  ART  1.1:  SUPPORT  TO  SITUATIONAL  UNDERSTANDING 


ART  1.3. 3.1  CONDUCT  A  ZONE  RECONNAISSANCE 


ART  1.3. 3. 2  CONDUCT  AN  AREA  RECONNAISSANCE 


ART  1.3. 3. 3  CONDUCT  A  RECONNAISSANCE-IN-FORCE 


ART  1.3. 3. 4  CONDUCT  A  ROUTE  RECONNAISSANCE 


ART  1.3. 3. 5  CONDUCT  A  RECONNAISSANCE  PATROL 


ART  1.3.4  CONDUCT  SURVEILLANCE 


Chapter  2  ART  2.0:  The  Maneuver  Battlefield  Operating  System 


ART  2.2.2  CONDUCT  ACTIONS  ON  CONTACT 


ART  2.2.3  EMPLOY  COMBAT  PATROLS 


ART  2.2.4  CONDUCT  COUNTERAMBUSH  ACTIONS 


iT  2.2.5  EXPLOIT  TERRAIN  TO  EXPEDITE  TACTICAL  MOVEMENTS 

4  t  y  /  Pi-in  rrh/  M  shriv  /  TTYT  Al  IT!  ft  All  PI  ATFOPMC!  \  Al  IITI  1PJFI  TcLre  ft.  All  PPc  /  IMR  Tcl^c  A  Ml  11  fVX 


UAV  A160  Hummingbird 


H  I 


x 

/  Min  iNn  Tack  ft  nrt;  /  am  pi  atformc  I  <  I  liiiiiiiiiiiii  ►  I 


Ml  13  A3 


Matching  Components  to  Tasks 


SLAD  Demo  Tasksf  H+90  to  H+98) 


Task  Name 


1 

2 

3 

4 

5 

6 

7 

8 

9 

10 
M 
12 

13 

14 

15 

16 

17 

18 
19 
*0 
El 

>2 

13 

M 

15 

m 

u 

m 

>9 

30 

31 

32 

33 

34 

35 


-  SLAD  DEMO  PROJECT 


□  H+90  to  H+92 


LSI  A1. 5.2  Occupy  an  Attack/Assault  Position  (AF 


LSI  A2.3.1  Collect  Relevant  Information  (ART  7.2.1 


LSI  A4.2.2  Conduct  Surveillance  (ART  1.3.4) 


LSI  A1. 6.2. 1.1. 4. 3  Report  Enemy  Information 


LSI  AS.  1.2  Detect  and  Locate  Surface  targets  (AR' 


LSI  A1. 6.2. 1.2  Conduct  UAV  Ops 


LSI  A4.2.1  Conduct  Tactical  Reconnaissance 


Duration 


0.38  days 


0.08  days 


1  hr 


2  hrs 


2  hrs 


2  hrs 


2  hrs 


2  hrs 


2  hrs 


Start 


Mon  9/6/04 


Mon  9/6/04 


Mon  9/6/04 


Mon  9/6/04 


Mon  9/6/04 


Mon  9/6/04 


Mon  9/6/04 


Mon  9/6/04 


Mon  9/6/04 


Finish 


Mon  9/6/04 


Mon  9/6/04 


Mon  9/6/04 


Mon  9/6/04 


Mon  9/6/04 


Mon  9/6/04 


Mon  9/6/04 


Mon  9/6/04 


Mon  9/6/04 


Predecessor: 


Resource  Names 


MCS.ARV.NLOS.Ci 


C2V 


UAV.ARV 


UAVrARV 


UAV 


UAVrARV 


3  4  5  6  7  8  9  10  11  12  1  2  3  4  5  6  7 


MCS,ARV,NLOS,C2V 

m  C2v 

ii  UAV.ARV 


UAV,ARV 
UAV 

I  UAV,ARV 


36 

D 


LSI  A1. 6.2. 1.2.1  Launch  and  Recover  U4V- 


Mnn  QfF.fr\A 


i  i aw  rvir-g 


LSI  A1. 6.2. 1.2.2  Fly  UAV  Mission 


-1  H+92  to  H+94 


Summary  Task  Information 


-3  LSI  A1.2  Conduct  Tactical  Maneuve 


LSI  A1.2.1.1  Employ  Travelling  Mov 


LSI  A1.2.1.2  Employ  Travelling  Gve 


LSI  A1.2.3.3  Exploit  Terrain  to  Expr 


LSI  A1.2.4.7.3  Negotiate  a  Tactical 


LSI  AS.  1.2  Detect  and  Locate  Surface  t 


LSI  A1. 6.2. 1.2  Conduct  UAV  Ops 


B  H+94  to  H+96 


MTP  17-5-2500. 17-KCRW  Occupy  a  V( 


MTP  17-5-2510. 17-KCRW  Occupy  a  Vf 


MTP  17-5-0011. 17-KCRW  Establish  an 


MTP  07-1-1  COP. 07-C332  Establish  the 


LSI  X  Disseminate  the  Common  Operat 


LSI  A1 .2.1 .2  Employ  Travelling  Gverwat 


LSI  A1 .2.3.3  Exploit  Terrain  to  Expediti 


LSI  A1.5.2  Occupy  an  Attack/As  sail  It  F 


LSI  X  Employ  Networked  Fires  to  Acce 


MTP  07-1-3000. 07-C3 32  Employ  Fire  S 


LSI  A5.1.2  Detect  and  Locate  Surface  t 


LSI  A1. 6.2. 1.2  Conduct  UAV  Ops 


-1  H+96  to  H+98 


LSI  A5.1.2  Detect  and  Locate  Surface  t 


LSI  A1. 6.2. 1.2  Conduct  UAV  Ops 


MTP  07-1-WT06.07.C332  Conduct  Batt 


General 


Predecessors 


Name:  |  LSI  A1.2  Conduct  Tactical  Maneuver  (7 

Predecessors: 


Duration:  j  0.08d  ^  |”  B 


ID 

Task  Name 

Type 

Lag 

9 

LSI  A4. 2.1  Conduct  Tactical  Reconnaissance 

Start-to-Start  (SS) 

lh 

3 

Help 


OK 


Cancel 


I  L-Juklhi] 


2  Fils  Muli  U/b/U4  Muh  y/b/U4 


■w 


H<l 


■V,MCS 

\f,C2V 


|  Resources 

Advanced 

Notes 

Custom  Fields  |  1 

□  L  j 


ICS 

ICS 

ICS,ARV,NLOS,C2V 
C  S,ARV,NLO  S,C2V 
AV,ARV 
AV 


MCS,NLOS 

m  mcs 
n  am 

n  C2v 

C2V 

m  mcs 

n  MCS,ARV,NL< 
MCS,HLOS,C2V 
n  NLOS 
m  NLOS 
n  UAV,ARV 

m  uav 


UAV,  A I 

UAV 

C2V 


•  Test  completed  mission  threads  in  a  live,  virtual,  constructive,  or  combination  environment.  For 
example,  DRC  and  Northrop-Grumman  SMEs  worked  together  to  develop  a  “what  if’  simulation 
model,  built  using  the  EXTEND  tool,  that  used  products  developed  during  steps  1-9  along  with  a 
Decision  Making  Process  flow  chart  to  execute  the  following  processes: 

•  Event  generation 

•  Component,  Sub-system,  Capability  State  changes 

•  Determine  platform  task  status,  collective  task  status,  mission  task  status 

•  And  produce  the  following  displays  and  outputs: 

•  Current  platform  capability 

•  Current  platform  task  ability 

•  Current  collective  task  status 

•  Current  mission  assessment  status 

•  Record  of  state  changes,  task  status  changes,  mission  health  status  changes 


Decision  Making  Process  Flow  Chart 


Mission  status 
goes  red 
Cannot  Accomplish 
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EXTEND  Tool  Use  to  Construct 
Mission  and  Means  Capability 
Degradation  Model  (M2CDM)  to: 

•  Map  component-level  state  changes  to  platform 
level  capabilities 

•  Map  platform  level  capabilities  to  ability  to 
perform  tasks 

•  Map  ability  to  perform  tasks  to  mission  success 


79 


Initialization  data  requirements  for  M2CDM 


Tactical  Scenario 

Military  Organizations  and  Equipment 
Important  Data  Elements 

-  Platform  to  Task  Relationships 

-  Sequence  of  Events 

-  Effect  of  Component  Loss  on  Platform  Capability 

-  Effect  of  Degraded  Capability  on  Task  Accomplishment 


TOP  LEVEL  FLOWCHART 
MMF  PROOF  OF  CONCEPT  MODEL 

MODEL  PURPOSE:  Does  the  Mission  Capability.  Package  Meet  the  Mission  Capability.  Requirement? 
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Missions  and  Means 
Degraded  Capabilities  Model  (M2DCM) 
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Mission  Readiness 


Mission 

Health 


f 


Mission  Health  Criteria 
Evaluates  status  of  collective  tasks. 

Mission  =  Green  if  all  Collectives  =  Green 
Mission  =  Yellow  if: 

One  Currently  Critical  Collective  is  Yellow,  or 
Two  or  more  Collectives  are  Yellow 
Mission  =  Red  if 

A  Collective  is  Red  and  Currently  Critical 
Two  or  more  Collectives  are  Red 


ART  2.4  Conduct  Direct  Fires 
ART  7.2  Manage  Tactical  Information 

Collective  Task  Health 

Green  =  Fully  Capable  Yellow  =  Acceptable  IDegradatio 
Red  =  Unacceptable  Degradation 


o  ART  1 .3  Conduct  Intell,  Surv  &.  Recce) 
o  LSI  A1 .2  Conduct  T actical  Maneuver 
o  ART  2.5  Occupy  an  Area 
o  ART  3.3  Employ  Fires 


Missions  and  Means 

drc  Degraded  Capabilities  Model  (M2DCM) 


Current  Tactical 
Action 


Capability  State 
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-■"BBBBBBBBB 
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y 


Capability  State 


MCS1 

MCS2 

MCS3 

MCS4 

MCSS 

MCS6 

MCS7 

MCSS 

MCS9 

Mobility 

Firepower 

Target  Acquisition  Direct  Fire 

Surveillance  &  Recce 

Communications 
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Crew 

Other  Mission  Functions 

Catastropic  loss 

Current  Tactical 
Action 


Platform  Capability  Roll  Up 

Green  =  Fully  Capable  Yellow  =  Acceptable  Degradation  Red  =  Unacceptable  Degrada 
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MCS3 
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MCS7 

MCSS 

MCS9 

Surface  to  Surface  Attack 

Disseminate  COP 

Conduct  Tactical  Maneuver 

Report  Enemy  Information 

1  1 

1  1 

Launch  and  Recover 

Fly  UAV  Mission 

Conduct  Recce 

Conduct  Surveillance 

Detect  and  Locate  Surface  Targets 

- 

Establish  the  COP 

Establish  and  Maintain  Commc 

Traveling  Movement  Technique 

Exploit  Terrain 

Negotiate  a  Tactical  Area 
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Task  Capability 


Fly  UAV  Mission 
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1 

1 
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1  1 

L 
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Collect  Relevant  Information 

1 _ 1 

1 _ 1 

1 _ 1 

□ 

□ 

□ 

1  1 

l  1 
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Employ  Fire  Support 

1  1 

1  1 

1  1 

1  1 

1  1 

1  1 
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Conduct  Lethal  Direct  Fire 

J  1 

1  i 

j  I 
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1  i 
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I  1 
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Platform  Task  Ability 
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Status 
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Red  Unit  Status 
White  =  Not  Detected 
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Missions  and  Means 
Degraded  Capabilities  Model  (M2DCM)  ahsaa 


Status  &  Recent 
Engagements 


T  as k^^ap ability 


28  Oct  05 


•  Application  of  the  previous  ten  steps  of  the  process  results  in  sufficiently  rigorous 
specification  of  the  mission  to  gain  meaningful  data  from  execution  in  LVC 
environment 

•  This  is  particularly  useful  for  evaluating  Effectiveness,  Suitability  and  Survivability 

•  Challenge  is  finding  the  appropriate  execution  environment  that  provides  a  sufficient 
level  of  granularity  and  fidelity  to  permit  observation,  evaluation  and  assessment  of 
tasks  against  MOEs  and  MOPs.  Execution  plan  may  drive  requirements  for  models 
and  simulations  not  currently  available 
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Lines  from  the 
TOEL 


TASKS 


PLATFORM 


0200-1000 

ART  7.2 

*MTP  17-5-001 1.17-KCRW  Establish  and 
Maintain  Communications 

ARV  2 

0412-0417 

ART  7.2 

*LSI  A1. 6. 2. 1.1. 4. 3  Report  Enemy 
Information 

ARV  2 

0200-1000 

ART  7.2 

*MTP  17-5-001 1.17-KCRW  Establish  and 
Maintain  Communications 

ARV  3 

0757-0802 

ART  7.2 

*LSI A1 .6.2.1 .1.4.3  Report  Enemy 
Information 

ARV  3 

0200-1000 

ART  7.2 

*MTP  07-1  -ICO P . Q7-C332  Establish  the 

C2V 

0200-0205.  0253-0253, 
0303-0313,  0341-0343, 
0437-0442,  0525-0530, 
0633-0638,0707-0712, 
0300-0305,  0349-0354 
0200-1000 

0200-1000 

0200-1000 

0255-0300,0313-0318, 

0339-0344,0410-0415, 

0523-0528,0612-0617, 

0706-0711,0750-0755, 

0344-0349 


ART  7.2  'ART  7.2.5  Disseminate  Common  Operational 
Picture  and  Execution  Information 


C2V 


0210-0542 

ART  7.2 

*MTP  17-5-001 1.17-KCRW  Establish  and 
Maintain  Communications 

UAV 1 

0250-0255,0305-0310, 

030-0335 

ART  7.2 

*LSI  A1 .6.2. 1 . 1 .4.3  Report  Enemy 
Information 

UAV 1 

0340-0835 

ART  7.2 

*MTP  17-5-001 1.17-KCRW  Establish  and 
Maintain  Communications 

UAV  2 

0431-0436,0715-0720 

ART  7.2 

*LSI  A1. 6. 2. 1.1. 4. 3  Report  Enemy 
Information 

UAV  2 

0543-1000 

ART  7.2 

*MTP  17-5-001 1.17-KCRW  Establish  and 
Maintain  Communications 

UAV  3 

ART  7.2 

LSI  A2.3.1  Collect  Relevant  Information  ART 

7.2.1 

C2V  ^ 

ART  7.2 

MTP  07-1 -WT06 . 07-C332  Conduct  Battle 
Tracking 

C2V 

ART  7.2 

*MTP  17-5-001 1.17-KCRW  Establish  and 
Maintain  Communications 

C2V 

ADT  11 

*1  ^1  A1  0  'I  1  A  1  Popnrt  Fnomy  Infm-matinri 

C1\! 

±1M''  TIMFI  IMFC  /  P-lin«  /  / 


MMF  Formal  Process  Diagram 


Outline 


Missions  and  Means  Framework 
Model 

11  Fundamental  Elements:  7  levels,  4  operators 


6.  Context,  Environment  (Military,  Civil,  Physical,  etc.) 


Developed  by  Dr.  Paul  Deitz,  Technical  Director, 
U.S.  Army  Materiel  Systems  Analysis  Activity  and 
Mr.  Jack  Sheehan,  Chief  Engineer,  Future  Combat 
Systems  Combined  Test  Organization 


•  Introduction 

•  Background/MMF  Review 

•  Using  MMF  to  get  to  the  “So 
What” 

•  Practical  application  -  How  MMF 
can  be  applied  to  ask  the  right 
questions  and  organize  the 
resulting  information. 

•  A  few  examples  of  MMF 
application  to  T&E 

•  Conclusion/Summary 


Planning 

Employment 
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MMF  T&E  Example 

TJE  Brief 
FCS  CTO 
(PLACEHOLDER) 


A 

1 

Level  3  Required  Capabilities  to  Train-JCAS  JCID 

2 

Battle-space  Awareness. 

3 

4 

Provide  for  the  development,  maintenance  and  dissemination  of  Common  Operating  Picture  [COP]  X 

5 

Integrate  LVC  data  into  the  Common  Operational  Picture  (COP] 

6 

Provide  for  ability  to  view  and  collect  data  from  the  Common  Operational  Picture  (COP]#,  V. 

7 

Develop  and  Maintain  Battle  Space  Awareness  via  Common  Operating  Picture  (COP]  \ 

S 

9 

Ensure  key  C2  nodes  such  as  JFACC,  JFLCC,  CAOC,  ASOC,  TACC,  DASC  have  adeguate  communications  systems  (including  secure  comms]  #  A  ^  \  .  * 

Integrate  range  COP  and  data  collection 

10 

Provide  for  target  location  at  all  levels  of  visibility^  A  ^ 

11 

Provide  for  target  designation  at  all  levels  of  visibility^  X 

12 

Provide  for  collaborative  mission  planning^ 

13 

Provide  for  collaborative  mission  rehearsal 

14 

Provide  for  positive  target  identification  at  all  levels  of  visibility  ^  ^ 

15 

\  / 

Provide  training  situations  for  practice  of  Battle  Damage  Assessment  (BOA]  and  integration  of  BOA  with  other  data  for  overall  comba\  vj 

16 

Ensure  timely  coordination  and  integration  of  Battle  Damage  Assessment  (BOA]  reports  from  aircrew  and  venue  generated  sources# 

17 

Provide  for  JCAS  Battle  Damage  Assessment  (BOA] 

IS 

Provide  virtual  and  constuctive  training  to  improve  battlefield  situational  awareness  of  entities  (friend,  foe,  neutral  or  other  non-combatant]  in  the  tactical  battlespace  and  target 
location  data,  including  updating  vehicles  and  other  entites  with  correct  signatures 

19 

20 

Command  and  Control. 

21 

22 

Mandate  and  facilitate  the  use  of  automated  systems  for  C2  by  BCT's  (l.e. Command  and  Control  Personal  Computer  (C2PC]  or  Force  Battle  Command  Battalion^Brigade  and 
Below  (FBCB2]#] 

23 

Mandate  exercise  planning  takes  into  account  peacetime  training  constraints  (location  restrictions,  airspace,  weapons  availability.availability  of  C4I  systems,  etc  ]  to  minimise  the 
impact  on  training.# 

24 

Provide  or  establish  Joint  Force  command  and  control  (C2]  structure  for  events 

25 

Provide  training  in  forward  area  Command  and  Control  (Cz]  of  aircraft 

26 

Ensure  ATO  and  SPINS  concur  for  C4  capabilities# 

27 

Mandate  and  facilitate  C2  and  eguipment  reguirements  are  addressed  for  training  and  observer  monitoring:# 

29 

Provide  complete  air  picture  at  command  nodes  (CAOC]# 

29 

Provide  for  Joint  Close  Air  Support  (JCAS]  Command  6c  Control  [C2]  needed  to  synchronise  JCAS  and  ground  maneuver 

30 

Facilitate  training  in  coordinating  airspace  for  UAV  support#,^ 

31 

Provide  for  constructive  command  nodes  to  provide  all  appropriate  orders  and  instructions  (comm  plan,  ATO,  SPINS,  etc.]# 

32 

Reguire  exercise  communications  architecture  replicate  real  world  operational  architecture# 

33 

Provide  for  information  exchange  via  interoperable  communications  (visual  i  voice  i  digital]^ 

34 

Enable  Have  Quick  Interoperability# 

35 

Mandate  Commander  development  of  Annex  K  for  Op  Order# 

36 

Provide  ability  for  OC's  to  access  key  communications  and  coordination  nodes#,  V* 

37 

Provide  for  communications  between  range  sites;  CAOC-N  and  NTC  air  assets,  Battlefield  Coordination  Detachment  (BCD]  # 

39 

Mandate,  facilitate  and  Provide  for  the  use  of  Joint  Data  Network  (JON]  IAV  doctrine  and  TTP# 

39 

Facilitate  training  in  Airspace  Coordination  Area  activation  and  deactivation  between  Division  Tactical  Operations  Center  (DTOC],  BCT,  and  Direct  Support  artillery  battalion 
TOCs#,^ 

40 

Provide  for  real-time  information  exchange  via  interoperable  communications  (visual  i  voice  i  digital] 

41 

Establish  Joint  training  strategy 
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ST  3.1.2  Assign  Joint/Multinartional  Theater  Firepower  to 
Targets/Target  Systems 

H 

X 

X 

X 

X 

X 

X 

X 

3 

ST  3.2.3  Synchronize  Theater  Strategic  Firepower 

H 

X 

X 

X 

X 

X 

4 

OP  2.2.5  Collect  Target  Information 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

5 

OP  2A2.4  Provide  Target  Intelligence  for  the  Joint 
Operations  Area  (JOA) 

H 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

6 

OP  3.1.1  Establish  Joint  Force  Targeting  Guidance 

X 

7 

OP  3.1.3  Develop  Operational  Targets 

X 

X 

X 

X 

3 

OP  3.1.5  Publish  Air  Tasking  Order(s)  (ATO) 

X 

X  

3 

OP  3.1.8  Coordinate  Immediate  Targets  for  Two  or  More 
Components 

H 

X 

X 

X 

X 

X 

X 

X 

X  ^ 

10 

OP  3.2.1  Provide  Close  Air  Support  Integration  for 

Surface  Forces 

H 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X  ^ 

11 

OP  3.2.7  Synchronize  Operational  Firepower 

H 

X 

X 

X 

X 

X 

X 

X 

X 

X  ^ 

12 

OP  5.1.1  Communicate  Operational  Information 

H 

X 

X 

X 

X 

X 

X 

X  ^ 

13 

OP  5.1.11  Provide  Positive  Identification  of  Friendly 
Forces  Within  the  Joint  Operations  Area 

H 

X 

X 

X 

X 

X 

X 

X 

X  111 

14 

OP  5.3.1  Conduct  Operational  Mission  Analysis 

H 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X  ^ 

15 

OP  5  A3  Provide  Rules  of  Engagement 

H 

16 

OP  6.1 .3.1  Employ  Positive  Control  Measures 

X 

X 

X 

X 

X 

X 

X  ^ 

17 

TA  2.4  Disseminate  Tactical  Warning  Information  and 
Attack  Assessment 

H 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X  ^ 

IS 

TA  3.2.2  Conduct  Close  Air  Support 

H 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X  ^ 

19 

TA  3.3  Coordinate  Battlespace  Maneuver  and  Integrate 
with  Firepower 

H 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X  111 

20 

TA  6.5  Provide  for  Combat  Identification 

H 

X 

X 

X 

X 

X 

X 

X 

X  ^ 

21 

HTA  2.2.1  Collect  Target  Information 

X 

X 

X 

X 

X 

X 

X 

X  ^ 

22 

HTA  3.1.5  Conduct  Tactical  Combat  Assessment 

X 

X 

X 

X 

X 

X 

X 

X 

23 

HTA  3.2.2  Attack  Enemy  Land  Targets 

X 

X 

X 

X 

X  ^ 

24 

HTA  3.2.4  Suppress  Enemy  Air  Defenses  (SEAD) 

X 

X 

X 

X 

X  ^ 

25 

HTA  3.2.8  Conduct  Fire  Support 

X 

X 

X 

X 

X 

X  III 

26 

HTA  5.1.1  Communicate  Information 

X 

X 

X 

X 

X 

X 

X 

X  ^ 

27 

HTA  5.1.3.1  Maintain  and  Display  Tactical  Picture 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X  111 

26 

HTA  5.1.3.2  Maintain  and  Display  Force  Command  and 
Coordination  Status 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X  ^ 

HTA  5A3  Synchronize  Tactical  Operations  and  Integrate 
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WRC  Facilities 


Level  3  Required  Capabilities  to  Train 

Battlespace  Awareness. 

Provide  for  the  development,  mainten ance  and  dissemination  of  Common  Operating  Picture  (CDF)  mA 

Integrate  LVC  data  into  the  Common  Operational  Picture  (CDF) 

Provide  for  ability  to  view  and  collect  data  from  the  Common  Operational  Picture  (COF)#,  'A 

Develop  and  Maintain  Battle  Space  Awareness  via  Common  Operating  Picture  [COP] 


Ensure  key  C2  nodes  such  as  JFACC,  JFLCC,  CAOC,  ASOC,  TACC,  DASC  have  adequate  communications  systems  (including 
secure  comms]  # 

Integrate  range  COP  and  data  collection 

Frovide  for  target  location  at  all  levels  of  visibility^ 

Frovide  for  target  designation  at  all  levels  of  visibility^ 

Frovide  for  collaborative  mission  planning^ 

Frovide  for  collaborative  mission  rehearsal 
Frovide  for  positive  target  identification  at  all  levels  of  visibility 

Provide  training  situations  for  practice  of  Battle  Damage  Assessment  (BDA)  and  integration  of  BDA  with  other  data  for  overall 
combat  assessment 

Ensure  timely  coordination  and  integration  of  Battle  Damage  Assessment  (BDA)  reports  from  aircrew  and  venue  generate^ 
sources# 

Provide  for  JCA5  Battle  Damage  Assessment  (BDA) 

Provide  virtual  and  constuctive  training  to  improve  battlefield  situational  awareness  of  entities  (friend, 
combatant)  in  the  tactical  battlespace  and  target  location  data,  including  updating  vehicles  and  other  el 
signatures 


NTC.  FI 
Irwin.  CA 


Nellis  AFB. 
NV 


MCAGCC. 
23  Palms. 
CA 


Naval 

Station  San 
Diego.  CA 


Command  and  Control. 


VSMR.  NM 


NAS  Fallon. 
NV 


Mandate  and  facilitate  the  use  of  automated  systems  for  C2  by  BCT's  (I.e.Command  and  Control  Personal  Computer  (C2PC)  or 
Force  Battle  Command  Battalion^Brigade  and  Below  [FBCB2]#) _ 

Mandate  enercise  planning  takes  into  account  peacetime  training  constraints  (location  restrictions,  airspace,  weapons 
availability.availability  of  C4I  systems,  etc  ]  to  minimize  the  impact  on  training.# _ 

Frovide  or  establish  Joint  Force  command  and  control  (C2)  structure  for  events _ 

Frovide  training  in  forward  area  Command  and  Control  (Cz)  of  aircraft _ 

Ensure  ATO  and  SFINS  concur  for  C4  capabilities# _ 

Mandate  and  facilitate  C2  and  equipment  requirements  are  addressed  for  training  and  observer  monitoring:# _ 

Frovide  complete  air  picture  at  command  nodes  (CAOC)# _ 

Frovide  for  Joint  Close  Air  Support  ( JCAS]  Command  &  Control  (C2)  needed  to  synchronize  JCAS  and  ground  maneuver _ 

Facilitate  training  in  coordinating  airspace  for  UAV  support#,^ _ 

Provide  for  constructive  command  nodes  to  provide  all  appropriate  orders  and  instructions  (comm  plan,  ATO,  SPINS,  etc.)# 
Require  exercise  communications  architecture  replicate  real  world  operational  architecture# _ 

Frovide  for  information  enchange  via  interoperable  communications  (visual  I  voice  i  digital]^ _ 

Enable  Have  Quick  Interoperability# _ 

Mandate  Commander  development  of  Annen  K  for  Op  Order# _ 

Frovide  ability  for  PC's  to  access  key  communications  and  coordination  nodes#,  'A _ 

Frovide  for  communications  between  range  sites;  CAQC-N  and  NTC  air  assets,  Battlefield  Coordination  Detachment  (BCD)  # 
Mandate,  facilitate  and  Provide  for  the  use  of  Joint  Data  Network  ( JDN)  IAW  doctrine  and  TTF# _ 

Facilitate  training  in  Airspace  Coordination  Area  activation  and  deactivation  between  Division  Tactical  Operations  Center 

rriTnn  RrT  andnirpcrRiinnnrrarHIlpmhari-aliririTnn^ttv: 


Outline 


Missions  and  Means  Framework 
Model 

11  Fundamental  Elements:  7  levels,  4  operators 


6.  Context,  Environment  (Military,  Civil,  Physical,  etc.) 


Developed  by  Dr.  Paul  Deitz,  Technical  Director, 
U.S.  Army  Materiel  Systems  Analysis  Activity  and 
Mr.  Jack  Sheehan,  Chief  Engineer,  Future  Combat 
Systems  Combined  Test  Organization 


•  Introduction 

•  Background/MMF  Review 

•  Using  MMF  to  get  to  the  “So 
What” 

•  Practical  application  -  How  MMF 
can  be  applied  to  ask  the  right 
questions  and  organize  the 
resulting  information. 

•  A  few  examples  of  MMF 
application  to  T&E 

•  Conclusion/Summary 


Planning 

Employment 
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Conclusion/Summary 


•  Top  -  Down  driven  Mission  to  Task  Decomposition 
Process  necessary  to  identify  tasks  required  to  accomplish 
the  Mission  -  Important  to  determining  ESS. 

•  Military  Professionals  use  doctrinal  processes  to  analyze 
Missions,  plan  Operations  and  monitor  and  assess  progress 
towards  accomplishing  Mission  during  execution. 

•  The  MMF  provides  a  disciplined  and  structured  framework 
that  translates  these  doctrinal  processes  into  a  detailed  data 
set  that  enables  a  common  framework  of  understanding 
between  stakeholder  communities 
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Missions  and  Means  Framework 


11  Fundamental  Elements: 


Seven  Levels,  Four  Operators 


7.  Why,  Wherefore,  to  What  End 
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Missions  and  Means  Framework 


11  Fundamental  Elements: 


Seven  Levels,  Four  Operators 


6.  Context,  Environmer 

it  (Mili 

tary,  Civil,  Physical,  etc.) 

7.  OWNFOR  Why  =  Purpose,  Mission 

7.  OPFOR  Why  =  Purpose,  Mission 

6.  Under  What  Circumstances 
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Missions  and  Means  Framework 


11  Fundamental  Elements: 


Seven  Levels,  Four  Operators 
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Missions  and  Means  Framework 


11  Fundamental  Elements: 


Seven  Levels,  Four  Operators 


6.  Context,  Environment  (Military,  Civil,  Physical,  etc.) 

7.  OWNFOR  Why  =  Purpose,  Mission  I  7.  OPFOR  Why  =  Purpose,  Mission 


n 


4.  Activity-centric,  named  with  a  Verb,  “Do  What”  —  the  Playbook 


102 


Missions  and  Means  Framework 


11  Fundamental  Elements: 


Seven  Levels,  Four  Operators 


6.  Context,  Environment  (Military,  Civil,  Physical,  etc.) 


2.  Entity-centric,  named  with  a  Noun  “By  Whom”  —  the  Players 
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Missions  and  Means  Framework 


11  Fundamental  Elements: 


Seven  Levels,  Four  Operators 


6.  Context,  Environment  (Military,  Civil,  Physical,  etc.) 


1.  The  (shared)  Slings-and-Arrows  of  Outrageous  Fortune  --  Science 
3.  Condition-dependent  “How  Well”  --  Engineering 
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Missions  and  Means  Framework 


11  Fundamental  Elements: 


Seven  Levels,  Four  Operators 


6.  Context,  Environment  (Military,  Civil,  Physical,  etc.) 


7.  OWNFOR  Why  =  Purpose,  Mission  I  7.  OPFOR  Why  =  Purpose,  Mission 


I 


% 

*♦4  iz^>  Planning 


J>  Employment 


Bottom-up,  Causal,  Time-forward 
execution  and  adjudication  of  outcomes 
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Missions  and  Means  Framework 


11  Fundamental  Elements: 


Seven  Levels,  Four  Operators 


6.  Context,  Environment  (Military,  Civil,  Physical,  etc.) 


7.  OWNFOR  Why  =  Purpose,  Mission  I  7.  OPFOR  Why  =  Purpose,  Mission 


n 


**•« 

Planning 

,.“rr . -'-J 

Employment 

Top-Down,  Concurrent 
Synthesis  and  Decision  Making 


106 


Missions  and  Means  Framework 


11  Fundamental  Elements: 


Seven  Levels,  Four  Operators 


6.  Context,  Environment  (Military,  Civil,  Physical,  etc.) 


7.  OWNFOR  Why  =  Purpose,  Mission  I  7.  OPFOR  Why  =  Purpose,  Mission 


n 


% 

Planning 


Employment 


Architecture  defines  how  Parts 
are  assembled  into  Packages 


Return 
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“Mo  More  Second  Opinions:  Organizing 
<Do(D  ModS  Within  the  Acquisition 

CycCe  ” 


James  (F.  O  ’(Bryon 
The  O  X Btyon  Cjroup 

443-528-2711 

NDIA  T&E  Conference, 
Jacksonville,  FL 

March  9,  2006 


M&S  AND  T&E 
ARE  PARTNERS, 
NOT  COMPETITORS 


MODELING  AND  SIMULATION 

IN 

TEST  6  EVALUATION 


Modeling  and  simulation  are  an  integral 
part  of  T&E  and  not  to  be  looked  at  as  a 
substitute,  nor  a  means  to  save  money. 
M&S  and  testing  are  mutually 
supportive  and  none  is  complete 
without  the  other. 


M  &  S  PLAY  A  VITAL  ROLE  EARLY  ON 
IN  SYSTEM  DESIGN  AND  VERIFICATION 

Source:  R.  Garrett,  “Opportunities  in  Modeling  and  simulation  to  Enable  Dramatic 
Improvements  in  Ordnance  Design,  “presented  to  the  Committee  on  Bridging  Design  and  Manufacturing. 
National  Research  Council,  Washington,  DC.,  April  29,  2003. 


SYSTEM  LIFE  CYCLE 


A  Proposal  That  Might  Work 
Program  Manager  Magazine 


JAMES  F.  O'BRYON 


In  the  following  few  pages,  I  discuss 
my  personal  thoughts  on  an  issue 
of  paramount  importance  not  only 
to  the  Department  of  Defense,  but 
also  to  the  nation’s  defense.  My 
hope  is  that  this  article  will  provoke  se¬ 
rious  thought  and  meaningful  action  to 
resolve  the  issues  raised. 

A  Lock  Bade 

Since  arriving  in  the  Pentagon  just  over 
12  years  ago,  and  for  more  than  a  decade 
before  that  serving  as  a  weapons  analyst 
in  the  Department  of  Defense  (DoD)  in¬ 
frastructure  away  from  the  Washington 
area,  I  have  been  witness  to  numerous 
and  surprisingly  similar  technical  and 
management  discussions  about  the  need 
to  get  the  modeling  and  simulation  ca¬ 
pabilities  of  the  DoD  organized,  incen- 
tivized,  under  control,  and  more  efficient 
to  better  serve  the  weapons  development 
and  acquisition  process. 

These  discussions  included  such  issues 
as  a  common  and  meaningful  model  ar¬ 
chitecture,  model  inter-connectivity,  lan¬ 
guage  consistency,  validation,  model 
proliferation,  and  configuration  control. 
They’ve  also  covered  the  problems  of  du¬ 
plication,  modeling  “stovepipes,”  the  lack 
of  meaningful  and  up-to-date  docu¬ 
mentation  supporting  M&rS,  and  of 
course,  the  lack  of  model  realism. 


O'Bryan  serves  as  the  Deputy  Director,  Operational 
lest  and  Evaluation,  Live  Fire  Testing,  in  the  Office 
of  the  Secretary  of  Defense,  The  Pentagon.  Wash¬ 
ington,  DC.  His  undergraduate  degree  is  in  Mathe¬ 
matics,  and  he  also  holds  two  graduate  degrees: 
one  in  Operations  Research  from  The  George 
Washington  University  and  another  through  the 
Electrical  Engineering  Department  of  the  Massa¬ 
chusetts  Institute  of  Technology 


TRUTH  #  1 


TRUTH  #  2 


“PMs  &  PEOs 
Control 
Largest 
Funding 
Blocks” 


TRUTH  #  3 


TRUTH  #  4 


“PMs  Have 
Short  Time 
Horizons 
And,  Hence, 
On  Their 
Investment, 
Decisions” 


TRUTH  #  5 


“Realistic 
M&S  Is  Not 
Necessarily 
Viewed  As  A 
Benefit  By 


The 


TRUTH  #  6 


“The  Golden 
Rule:  Them 
That  Have 
The  Gold, 
Make  The 
Rules” 


“MASTER” 


MODELING  AND 
SIMULATION  TEST  AND 
EVALUATION  REFORM 


What  is  ‘MASTER’? 


•  MASTER  is  a  management  approach  to 
modeling  and  simulation  in  support  to  the 
defense  department’s  policy  of  simulation- 
based  acquisition 

•  It  will  provide 

•  critical-mass  funding 

•  add  discipline  to  the  development  of 
modeling  and  simulation 

•  assure  that  funds  expended  on  modeling  and 
simulation  are  spent  to  further  the  state  of 
the  artj  including  VV&A 

•  add  connectivity  across  various  model 
vectors  being  developed 

•  free  up  the  Program  Manager’s  time  & 

concerns  about  modeling  and  simulation 
support 

•  assure  the  most  realistic  models  & 
simulations  are  exercised  in  designing 

testing,  evaluating,  training,  fielding  and 
fighting  our  systems. 


Consortium  Discussion 


Program  Managers  would  initially  describe  their 
system(s),  acquisition  strategy,  and  M&S 
requirements  to  a  consortium  which  would  then 
parse  out  these  needs  into  vectors  of  M&S  technical 
responsibility. 

Consortium  Members,  who  are  charged  with  having 
knowledge  of  state  of  the  art ,  as  well  as  where  it 
exists  within  and  outside  of  their  respective 
organizations,  would  make  the  decisions  as  to  which 
M&S  tools  best  suit  the  PM’s  needs  and  where  the 
funds  would  be  expended  to  meet  the  specific 
requirements  of  each  Program  Manager’s  system(s). 
They  would  upgrade  extant  models  where  available 
and  originate  M&S  only  when  absolutely  necessary. 
In  many  instances,  these  investments  would  be 
allocated  to  organizations  external  to  the 
Consortium  Membership  itself. 


Must  Have  Up-Front 
Investment  in  M&S!! 


“I  expect  programs  to  make  the  up-front 
investment  in  modeling  and  simulation 
application  technology,  and  will  be 
looking  for  evidence  of  that  investment 
in  program  planning  and  execution.” 


Honorable  Jacques  S.  Gansler,  Under  Secretary 
of  Defense  (Acquisition  and  Technology)  1998 


If  there’s  no  new  money,  where 
will  the  money  come  from  to 
fund  this  “MASTER”  initiative? 


Where  Would  Money  to  Fund  the 
Consortium  Come  From? 
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A  modest  tax  (“greens  fee”)  would  be  assessed  upon 
every  Program  Manager’s  total  budget.  These  funds 
would  be  placed  in  the  Consortium’s  account  to 
provide  the  needed  M&S  support  to  the  Program 
Manager. 

The  proposed  “tax”  would  be  a  percentage  of  the 
Program  Manager’s  budget  (perhaps  2-3%).  This  is 
significantly  less  than  what  is  currently  spent  by 
PM’s  on  a  plethora  of  isolated  M&S  activities. 

The  tax  would  not  be  at  the  discretion  of  the 
Program.  It  would  be  a  policy  decision  and 
implemented  early  on  at  the  OSD  Comptroller  level. 

Funds  would  be  removed  early  to: 

•  enable  sufficient  time  to  develop  the  needed  M&S, 
and, 

•  avoid  the  tendency  to  cut  the  funding  of  modeling 
and  testing  programs,  when  problems  arise  and 
budgets  get  tight 


Potential  Modeling  Vectors 
Needed  for  the  Testing  and 
Training  Communities 


SOME  EXAMPLES: 

TERRAIN 

WEATHER 

CADCAM  SYSTEM  DESCR 
AERODYNAMIC  FLOW/FLIGHT 
STABILITY 
6  DOF  FLY-OUT 
TARGET  SIGNATURES 
SENSOR/FUZING 
SMOKE/OBSCURATION 
C3I 
EW 

ACOUSTIC 
BALLISTIC 
1-1  ENGAGEMENT 
MxN ENGAGEMENT 
VULNERABILITY 
LOGISTICS 
MANY  OTHERS 


What  are  Some  of  the  Benefits 
of  Forming  a  Consortium  to 
Oversee  M&S  Investment 
Within  the  DoD? 


1.  It  would  assure  that  Program  Managers 

have  the  best  and  most  realistic  model 
support  for  their  programs. 

2.  It  would  establish  the  necessary  consortium 

protocols  for  model  architecture, 
languages  and  protocols,  insuring  that  no 
funds  would  be  invested  in  model 
development  or  upgrades  unless  they 
meet  these  protocols,  thereby  facilitating 
interoperability. 

3.  It  would  assure  that  model  investments 

would  be  directed  toward  extending  the 
capability  of  extant  models  and 
simulations  rather  than  spending 
significant  funds  reinventing  and  re¬ 
buying  codes  which  exist  or  exist  in 
part. 


Consortium  Benefits? 
(Continued) 


4.  The  structure  would  provide  an  adequate 

source  of  funding  to  extend  the  state  of  the 
art  in  the  M&S  base,  instead  of  being  at 
the  whim  of  the  Program  Manager, 
typically  trying  to  maximize  the  short¬ 
term  return. 

5.  It  would  focus  national  expertise  in  each 

technical  discipline  to  assure  that 
decisions  on  which  model  investments 
were  indeed  needed  in  each  of  these 
disciplines. 

6.  It  would  free  up  the  Program  Manager’s 

time  and  attention  to  other  management 
responsibilities  and  allow  the  Consortium 
to  provide  the  needed  M&S  support  for 
each  respective  program. 


March/ 


A  PUBLICATION  OF  THE  DEFENSE  ACQUISITION  UNIVERSITY! 


4  DoD’s  Technical 
Information  Broker 


Defense  AT&L  Interviews 
R.  Paul  Ryan,  Administrator 
mmmm  Defense  Technical 
Information  Center 

BlBlD1"rK”  ■ 

ALSO 


]  America’s  Imprudent 
4Vi  and  Unsustainable 

c?|2lE?r‘  i  ii .  Fiscal  Path 


David  M.  Walker, 
U.S.  Comptroller  General 


dig  And  Introducing 

Defense  AT&L’s  First  Graphic  Article 


DoD’s  Modeling  and  Simulation 
Reform  in  Support  of  Acquisitior 

Stop  Kicking  the  M&S  Can  Down  the  Road 


James  F.  O’Bryon 


as  being  among  its 
primary  methods  for 
reducing  time  to  mar¬ 
ket  for  defense  systems 
and  reducing  the  cost  of  these  sys 
terns  at  the  same  time.  The  fol¬ 
lowing  statement  is  contained 
in  a  letter  dated  March  21, 

2000,  addressed  to  the  Office 
of  the  Secretary  of  Defense, 

Service  secretaries,  the  De¬ 
fense  Intelligence  Agency, 
and  the  Joint  Chiefs  of  Staff; 
it  is  cosigned  by  the  under 
secretary  of  defense  (acqui¬ 
sition,  technology  and  logis¬ 
tics)  (USD(AT&L))  and  the  di¬ 
rector,  operational  test  and 
evaluation,  (DOT&E):  “We  have 

the  funds,  however,  are  spent  in  support  of  the  r 
development,  test,  and  evaluation  of  new  def 
quisition  programs. 

In  an  article  in  the  July  2005  issue  of  National 
Magazine,  David  W.  Duma,  the  Pentagon’s  actii 
tor,  operational  test  and  evaluation,  wrote  that 
fense  Department  needs  to  better  manage  its  sir 
programs.  I  think  we’ve  kind  of  lost  our  way 
partment  with  modeling  and  simulation.  Multij 
cies  are  buying  duplicate  technologies,  rather  th 
dinating  efforts.  We  are  using  more  model 
simulation.  But  it’s  not  focused,  it’s  scattered.  Ev 
is  building  their  own.” 

Not  a  New  Problem 

1  couldn’t  agree  more.  So  why  does  the  DoD  cor 
lose  its  way  using  more  M&S  but  in  a  “scattered 


stressed  that  we  must  make  better  use  of  modeling 
and  simulation  (M&S)  to  improve  the  acquisition 
process,  reduce  costs,  enhance  T&E  [test  and  evalua¬ 
tion],  and  shorten  development  times  for  our  new  sys¬ 
tems.  We  are  convinced  that  efficient  use  of  M&S 
throughout  the  system  life  cycle  will  net  great  dividends 
in  efficiencies.” 

Few  people  would  argue  that  M&S  is  not  an  important  el¬ 
ement  in  the  acquisition  process.  The  question  is  this: 
Has  there  been  progress  within  DoD  to  efficiently  orga¬ 
nize,  fund,  develop,  promulgate,  and  maintain  configu¬ 
ration  control  of  the  DoD’s  massive  and  diverse  M&S  ac¬ 
tivities  to  yield  the  efficiencies  so  clearly  stated  in  the 
letter  quoted  above?  Estimates  for  how  much  is  spent  an¬ 
nually  on  M&S  in  the  DoD  range  from  $5  billion  to  $30 
billion,  depending  on  how  one  defines  M&S.  Some  of  this 
is  spent  on  M&S  in  support  of  training.  The  majority  of 


odeling  and  simula¬ 
tion— M&S— has  long 
been  touted  by  the  De¬ 
partment  of 
Defense 


O’Bryon  served  as  deputy  director,  operational  test  and  evaluation  in  the  Office  of  the  Secretaiy  of  Defense  until  November  2001.  He  current 
as  a  consultant  to  ORSA  Corporation.  Aberdeen.  Md. 
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IF  YOU  HAVE  SOME  IDEAS  YOU'D  LIKE 
TO  SHARE  OR  WOULD  LIKE  TO 
CHALLENGE  SOME  O  THESE  IDEAS,  I 
WOULD  WELCOME  YOUR  IDEAS. 
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THE  O'BRYON  GROUP 

Consultants 

National  Defense  &  Security 

JAMES  F.  O'BRYON 


1608  South  Tollgate  Road  Voice  (410)  515-03 

Bel  Air,  MD  21015-5825  Fax  (410)  569-96 

iamesobrvon@obrvonaroLto.com  Cell  (4431 528-27 

Call  me  at  410-515-0345  op  email  me  at 

jamesobryon@obryon 


Modeling  &  Simulation 

How  Can  M&S  Team  Better  With  DT&E? 


Steven  E.  Cameron 

Senior  Manager,  Modeling  &  Simulation  Infrastructure 
Boeing  Analysis,  Modeling,  Simulation  &  Experimentation 

March  7th,  2006 

NDIAT&E  Forum,  Jacksonville,  FL 


Copyright  ©  2005  Boeing.  All  rights  reserved. 


Thoughts 


•  Industry-government  teaming — early  and  often 

•  Refinement  of  Live,  Virtual,  and  Constructive 
M&S  capability 

•  Industry  access  to  government  systems 

•  Organizational  support  of  M&S  in  DT&E 

•  Lifecycle  perspective 

•  Open  Standards 


M&S  isn't  a  nicety ,  but  a  necessity 
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Complexity 


Complexity  drives  the  need  for  M&S 


104  105  106  107  10s  109 

Number  of  Pieces  to  Integrate 


Complex  Systems  of  Systems  demand  M&S  in  development  process. 


Copyright  ©  2005  Boeing.  All  rights  reserved. 
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mmmM 


Roles  of  Modeling  &  Simulation 


Requirements 

Definition, 

Concept  Development, 
and 

Customer  Advocacy 


Training 


‘ 


Operational 
Test  &  Evaluation 


its  reserved. 
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Building  Block  SoS  Test  Approach 


Systems  simulated  to 
comprise  subset  of  SoS 


System  simulations 
upgraded  and  coupled 
with  early  emulations 
comprise  subset  of  SoS 
with  SoS  interfaces 


System  simulations  and 
emulators  upgraded  and 
coupled  with  prototype 
hardware  to  comprise  SoS 


Live/Virtual/Constructive 


Early  phases  are  simulation  and  emulation  based  to  gain  knowledge  for 
Design  &  Development  to  set  up  for  Test  success 


Copyright  ©  2005  Boeing.  All  rights  reserved. 
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Organization  Matters 


•  Recognizing  the  value  of  M&S,  Boeing 
recently  formed  an  overarching  modeling  & 
simulation  organization 

>  Integrates  organizations  and  functions  that  cross 
program  boundaries,  and 

>  Provides  analysis  of  network  enabled  systems  of 
systems 

>AII  connected  by  one  knowledge  base 

•  M&S  Infrastructure  formed  around  NCO 
Industry  Consortium  architecture 


Copyright  ©  2005  Boeing.  All  rights  reserved. 
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M&S  Architecture 


HUMAN  INTERFACE  x' 


APPLICATIONS  -< 


Models  and  Tools 


INFORMATION 


COMMUNICATION  V 


Copyright  ©  2005  Boeing.  All  rights  reserved. 
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Colorado  Springs! 

- 1 - w - ' 


Palmdale 


_ 


i  4  T Philadelphia 


St.  Louis 


(Rosslyn 


- 


Crystal  City 


Anaheim 


Mesa 


Long  Beach 


Huntsville 


Huntington  Beach 
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Three  Levels  of  Military  Simulation 


•  Operators 

•  ADV  NCO  C2 

•  BMC4ISR 

•  Interoperability 

•  SBR/Space  Assets 

•  UCAV/UAV 

•  SHORAD 

•  Weapons 


•  Operators 

•  F/A-18;  EA-18G 

•  Apache 

•  AWACS 

•  CRW 


Operators 
CONOPS/TTPs 
Systems-of-Systems 
Interoperability 
Tactical  S&R/C2/BM 


Copyright  ©  2005  Boeing.  All  rights  reserved. 
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Virtual  Warfare  Center 


65,000  square  feet 
23  SCIFs 

Dual  operational  capability 
Flexible  network  options 


Video-over-internet  capability 

Ability  to  simulate  more  than  2,000  digital  bodies 

Six  F/A-18  cockpits,  four  F/A-22  cockpits,  four  F-15  cockpits 

Simulate  such  platforms  as  AWACS,  AEGIS,  Patriot  and 
other  ISR  systems 

Various  patent  pending  simulation  technologies  to  expand 
scenario  sizes 


Copyright  ©  2005  Boeing.  All  rights  reserved. 
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Government  -  Industry  M&S  Connection 


Copyright  ©  2005  Boeing.  All  rights  reserved. 
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Live  &  Virtual  Data  Link  Integration  Demonstration  —  December  2005 


Copyright  ©  2005  Boeing.  All  rights  reserved. 
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Impediments 


•  Zealotry 

•  Different  goals 

>  Programmatic 
>DT&E 

•  Intellectual  Property 

•  Security 


Notice  the  lack  of  technological  impediments. 


A 
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Summary 


•  M&S  is  critical  for  successful  DT&E  of 
complex,  net-enabled  systems  of  systems 

>Requires  robust  application  of  M&S  across  the 
entire  product  life  cycle 

•  M&S  is  available,  today,  to  provide  credible 
DT&E  involving  Live,  Virtual,  and 
Constructive  elements 


Partnerships  are  key  -  both  government/industry,  and  industry/industry. 


A 
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Presentation  Outline 


■  M&S  and  T&E  Background 

■  Overview  of  Prior  Acquisition  M&S  Studies 

■  Selected  Study  Recommendations  for  M&S 
Related  to  T&E 

■  Some  Common  Themes  in  Acquisition  M&S 
Study  Recommendations 

■  Some  Personal  Observations  on  the  Way  Ahead 


T&E  and  M&S: 

Definition  Linkages 

Test  and  Evaluation  (T&E) 

Process  by  which  a  system  or  components  are  exercised  and  results  analyzed  to  provide 
performance-related  information.  The  information  has  many  uses  including  risk  identification 
and  risk  mitigation  and  empirical  data  to  validate  models  and  simulations.  T&E  enables  an 
assessment  of  the  attainment  of  technical  performance,  specifications,  and  system  maturity 
to  determine  whether  systems  are  operationally  effective,  suitable  and  survivable  for  intended 
use,  and/or  lethal.  ... 

Model 

A  representation  of  an  actual  or  conceptual  system  that  involves  mathematics,  logical 
expressions,  or  computer  simulations  that  can  be  used  to  predict  how  the  system  might 
perform  or  survive  under  various  conditions  or  in  a  range  of  hostile  environments. 

Simulation 

A  simulation  is  a  method  for  implementing  a  model.  It  is  the  process  of  conducting 
experiments  with  a  model  for  the  purpose  of  understanding  the  behavior  of  the  system 
modeled  under  selected  conditions  or  of  evaluating  various  strategies  for  the  operation  of  the 
system  within  the  limits  imposed  by  developmental  or  operational  criteria.  Simulation  may 
include  the  use  of  analog  or  digital  devices,  laboratory  models,  or  “testbed”  sites.  ... 


Source:  Glossary  of  Defense  Acquisition  Acronyms  and  Terms,  12th  edition,  Defense  Acquisition  University,  July  2005. 
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The  “V”  Process  Model  for 
Systems  Engineering 


Source:  Benjamin  Blanchard,  Wolter  Fabrycky,  Systems  Engineering  and  Analysis,  Prentice  Hall,  New  Jersey,  1998 
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One  View  of  the  Relationships  among 
Modeling,  Simulation,  Test,  and  Evaluation 


Source:  Simulation,  Test,  and  Evaluation  Process  (STEP)  Guidelines,  Director,  Operational  Test  and  Evaluation, 
and  Director,  Test,  Systems  Engineering  and  Evaluation,  4  Dec  1997. 

- dPL 


A  Decade  of  Acquisition  M&S  Studies  (1  of  2) 


1994  J  Final  Report  of  the  Acquisition  Task  Force  on  Modeling  and  Simulation  (DDR&E) 

N  Naval  Research  Advisory  Committee  Report  on  Modeling  and  Simulation 
( N  RAC/AS  N(RD  A)) 

1995  N  Collaborative  Virtual  Prototyping:  An  Assessment  for  the  Common  Support  Aircraft 

Initiative  (NAVAIR) 

1996  I  Collaborative  Virtual  Prototyping  Sector  Study  (North  American  Technology  and  Industrial 

Base  Organization) 

I  Study  on  the  Application  of  Modeling  and  Simulation  to  the  Acquisition  of  Major  Weapon 
Systems  (American  Defense  Preparedness  Association  (ADPA)) 

J  Study  on  the  Effectiveness  of  Modeling  and  Simulation  in  the  Weapon  System  Acquisition 
Process  (DoD  Director,  Test,  Systems  Engineering  and  Evaluation) 

1997  P  Technology  for  the  United  States  Navy  and  Marine  Corps,  2000-2035,  Becoming  a  21st 

Century  Force,  Volume  9:  Modeling  and  Simulation  (Naval  Studies  Board,  National 
Research  Council  (NRC)) 

J  Simulation,  Test,  and  Evaluation  Process  (STEP)  Guidelines  (DOT&E  and  DTSE&E), 
December  4, 1997 


■  Joint  DoD  (J)  ■  Navy-oriented  (N)  ■  Industry-driven  (I)  ■  Private  organization  (P) 

- dPL 
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A  Decade  of  Acquisition  M&S  Studies  (2  of  2) 

1998  J  A  Roadmap  for  Simulation  Based  Acquisition  -  Report  of  the  Joint  Simulation  Based 

Acquisition  Task  Force,  December  4, 1998 

J  Simulation  Based  Acquisition:  A  New  Approach  -  Report  of  the  Military  Research 
Fellows,  Defense  Systems  Management  College,  1997-1998 

1999  P  Advanced  Engineering  Environments:  Achieving  the  Vision,  Phase  1  (NRC  -  prepared 

for  NASA) 

2000  P  SIMTECH  2007  Mini-Symposium  and  Workshop  Proceedings:  Session  1,  December 

1997,  and  Session  2,  August  1998  (Military  Operations  Research  Society  (MORS)) 

2001  J  Results  of  the  DOT&E/LFT&E  Survey  of  Modeling  &  Simulation 

2002  P  Modeling  and  Simulation  in  Manufacturing  and  Defense  Systems  Acquisition: 

Pathways  to  Success,  June  2002  (NRC  -  prepared  for  Defense  Modeling  and 
Simulation  Office  (DMSO)) 

2004  I  M&S  Support  to  the  New  DoD  Acquisition  Process,  February  2004  (National  Defense 

Industrial  Association  (NDIA)  Systems  Engineering  Division,  Modeling  &  Simulation 
Committee) 

2005  J  Acquisition  Modeling  and  Simulation  Master  Plan  (Preliminary  Draft),  Acquisition  M&S 

Working  Group 


■  Joint  DoD  (J)  ■  Navy-oriented  (N)  ■  Industry-driven  (I)  ■  Private  organization  (P) 

- dPL 
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Naval  Research  Advisory  Committee: 
Distributed  Simulation  Based  Acquisition 


Data 

(Collected 

Empirical 

Knowledge) 


Model 

(Representation 
Of  Physical 
Phenomena) 


Computer  Aided 
Design, 
Engineering, 
Manufacture 


Advanced 

Distributed 

Simulation 


Simulation 
Based 
Design  / 
Manufacturing 


_ L 

Distributed  Simulation  Based  Acquisition 


Source:  Naval  Research  Advisory  Committee  Report  on  Modeling  and  Simulation,  NRAC  94-3,  Nov  1994. 
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Selected  Finding  /  Recommendation  from  Study 
on  Effectiveness  of  M&S  in  Acquisition  (1996) 


Study  on  the  Effectiveness  of  Modeling 
and  Simulation  in  the  Weapon  System 
Acquisition  Process 


October  1996 


Final  Report 


■  The  issue  is,  to  what  degree  can  M&S  replace  or  augment  field  tests? 

□  There  is  no  universal  answer,  but  the  message  received  in  part  by  the 
community  is  that  the  thought  and  guidance  are  there,  but  the  implementation 
and  acceptance  are  not. 

■  To  meet  the  challenge  of  institutionalizing  the  use  of  available  technology,  the 
Services  must  be  committed  to  providing  funds  for  modeling  and  simulation 
at  the  inception  of  the  program. 

□  OSD  and  the  Services  should  commit  Science  and  Technology  dollars  to 
upgrade  capabilities  and  facilities  that  could  serve  many  weapon  system 
acquisitions. 

□  Program  managers  should  be  encouraged  to  use  these  facilities  and 

capabilities  instead  of  contracting  to  have  their  own  system  specific  facilities 
and  tools  built. 


Background  /  Motivation  for  Several  DoD  Acquisition 
M&S  Studies  in  the  late  1990s 


DoD  Simulation  Based  Acquisition 
Vision  Statement  and  Goals 

Vision:  An  Acquisition  Process  in  Which  DoD  and  Industry  Are  Enabled 
by  Robust,  Collaborative  Use  of  Simulation  Technology  That  Is 
Integrated  Across  Acquisition  Phases  and  Programs. 

Goals:  The  Goals  of  Simulation  Based  Acquisition  (SBA)  Are  to: 

-  Substantially  Reduce  the  Time,  Resources,  and  Risk  Associated  With 
the  Entire  Acquisition  Process; 

-  Increase  the  Quality,  Military  Worth  and  Supportability  of  Fielded 
Systems,  While  Reducing  Total  Ownership  Costs  Throughout  the  Total 
Life  Cycle; 

-  Enable  Integrated  Product  and  Process  Development  (IPPD)  Across  the 
Entire  Acquisition  Life  Cycle. 

Source:  Acquisition  Council  of  the  DoD  Executive  Council  for  Modeling  and  Simulation  (approved  5  Dec  1997). 
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SIMULATION,  TEST,  AND 
EVALUATION  PROCESS 


Some  Sound  Bites  from  the  STEP  Guidelines: 

GUIDELINES 

M&S  and  the  TEMP 

4  DECEMBER  1997 


The  TEMP  should 

■  document  how  the  early  use  of  M&S  will  aid  in  assessing  vulnerability  and  lethality 

■  address  STEP  resources  that  will  help  identify  areas  where  data  is  needed  and  where  M&S 
development  is  needed 

■  identify  areas  where  actual  testing  either  can  be  augmented  by  M&S  or  used  to  validate  the 
models  and  simulations 

■  summarize  the  M&S  VV&A  and  the  data  certification  to  be  conducted 

■  reflect  the  integration  of  models,  simulations,  and  test  events  to  obtain  the  most  credible  data 
with  which  to  conduct  a  comprehensive  evaluation  of  performance 

■  include  a  discussion  of  the  mission-level  and  engagement-level  models  and  simulations,  that  will 
be  used  to  identify  COIs  and  the  operationally  significant  MOEs  and  MOPs,  so  CTPs  can  be 
derived 

■  document  how  the  early  use  of  M&S  will  aid  in  assessing  the  operational  impact  of  suitability 
issues  (the  "ilities")  and  logistics,  and  in  determining  initial  tactics,  training,  and  procedures 

■  document  the  integrated  use  of  accredited  models  and  simulations  with  OT  to  increase  the 
knowledge  and  understanding  of  the  capabilities  and  the  limitations  of  the  system  as  it  will  be 
employed 

■  include  the  resources  required  to  VV&A  the  models  and  simulations;  the  resources  required  to 
obtain,  maintain,  and  M&S;  and  the  resources  required  to  archive  data  for  the  M&S 


rtPL 
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Some  Recommendations  from 
the  SBA  Roadmap  (1998) 

■  Short  Term: 


A  limn!  Mh|i  for  Simulation  Itowil  AajiiiMliim 

Ki'pitrl  iiT TIu1  Joint  Simuhiliim  liwd  Aiquisitkii  Tiiik  liim1 


□  The  T&E  community  should  develop  additional  Deskbook  practices  addressing  the 
specifics  and  actual  examples  of  M&S  integrated  with  T&E  in  the  evaluation  strategy  (layer 
of  detail  more  specific  than  Simulation,  Test,  and  Evaluation  Process  (STEP) 
Guidelines). 

•  Criteria  for  accepting  verification  and  validation  (V&V)  for  models  to  be  accredited  by 
the  T&E  community  should  be  addressed 

•  Practices  and  procedures  should  also  include  techniques  regarding  the  pooling  of 
test  data  with  simulation  data,  and  associated  limitations. 

□  Continue  to  fund  and  support  development  of  virtual  T&E  infrastructure  to  include 
synthetic  test  environments,  virtual  firing  ranges  and  proving  grounds,  etc. 

■  Long  Term: 

□  Review  OT  and  Live  Fire  Testing  policies  with  respect  to  M&S  as  technological 
advancements  occur  (and  M&S  capabilities  improve)  to  determine  if  changes  to 
legislation  should  be  proposed. 

□  Continue  efforts  to  evolve  and  advance  virtual  T&E  infrastructure. 


rtPL 
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Selected  Recommendations  from  DOT&E  / 
LFT&E  Survey  of  Modeling  &  Simulation  (2001) 


HAI 

Results  of  the  DOT&E/LFT&E  Survey  of 
Modeling  &  Simulation 

presented  at  the 

Acquisition  Functional  Working  Group 

2  May  2001 

Anne  Hillegas 

1710SA1C  Dr.  -  Suit.  «°° '  FfX™e76!*'3  ' 

■  Emphasize  the  important  role  that  acquisition  programs  must  play  in  the 
development  of  M&S 

□  address  M&S  in  the  5000  series 

□  incentivize  Program  Office  investment  in  M&S 

■  Foster  an  improved  understanding  of  the  interrelationship  of  T&E  and 
M&S 

□  Endorse  pilot  programs  with  the  SAEs  that  examine  and 
demonstrate  the  utility  of  M&S  for  T&E 


Some  Sound  Bites  from  the  NRC  M&S  in 
Manufacturing  and  Acquisition  Study  (2002) 

■  “It  will  therefore  be  necessary  to  create  and  sustain  an  acquisition  infrastructure,  including 
an  M&S  infrastructure.  DOD  acquisition  personnel  could  use  M&S  to  predict  the  cost- 
effectiveness  of  potential  solutions,  thereby  reducing  the  need  to  produce  and  test 
expensive  hardware  prototypes.” 

■  “The  desire  to  achieve  and  maintain  interoperability  requires  early  and  continuing 
commitment  to  several  orchestrated  activities.  These  include  development  of  common 
standards,  protocols,  and  data  definitions;  agreed-upon  concepts  of  operation;  testing 
and  evaluation  to  ensure  that  agreed-upon  actions  have  been  implemented  properly;  and 
configuration  management  of  systems  to  assure  proper  management  of  evolutionary 
changes.” 

■  “Task  Force  XXI  demonstrated  two  important  acquisition  needs:  first,  the  significance  of 
coevolving  the  system-of-systems  with  continual  dialogue  among  all  major  stakeholders; 
and  second,  the  need  for  a  virtual  M&S  testbed  to  enable  this  dialogue.” 

■  “As  M&S  is  used  more  in  operational  testing,  the  demands  on  the  validation  of  the 
simulation  will  increase.  ...  Validation  methods  that  quantify  the  bounds  of  validity  and 
risk  of  error  in  a  model  can  help  to  establish  the  limits  of  M&S  applicability  in  operational 
test  and  evaluation.” 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Selected  Findings  from  the  NDIA  Systems 
Engineering  M&S  Committee  Report  (2004) 

■  Distributed  simulation  offers  a  practical,  cost-effective  way  to  integrate  and  test  systems 
of  systems. 

□  Rounding  up  all  the  actual  hardware  to  compose  an  entire  system  of  systems  for 
integration  and/or  test  becomes  increasingly  impractical  as  the  number  of  systems 
in  an  SoS  grows. 

■  Substituting  credible  simulations  for  the  real  weapon  systems  not  under  test  offers  an 
optimal  way  to  integrate  individual  systems 

□  It  is  flexible,  scalable,  and  protects  proprietary/intellectual  property. 

□  The  capability  to  intermix  simulations,  real  systems,  and  lab  assets  (“hardware  in  the 
loop”)  in  a  distributed  simulation  federation  has  been  repeatedly  shown  to  be  both 
feasible  and  helpful 

■  A  promising  example  of  the  above  approach  is  the  Joint  Distributed  Engineering  Plant 
(JDEP),  but  it  is  currently  under-resourced. 

□  A  non-proprietary,  tailorable,  reasonably-available  family  of  such  federations  would  go 
a  long  way  toward  providing  the  ‘standard’  environments  needed  to  explore  the  time- 
coordinated,  dynamic  interactions  of  an  SoS. 

■  The  existence  and  adoption  of  open  standards  can  allow  M&S  tools  to  be  flexibly  used 
together  to  address  the  challenges  of  the  new  acquisition  process. 

rtPL 


National  Defense  Industrial  Association 
Systems  Engineering  Division 
Modeling  &  Simulation  Committee 

Study  Task  Report 

M&S  Support  to  the  New  DoD  Acquisition  Process 


15 


Some  Recommended  Actions  from  the  Acquisition 
M&S  Master  Plan  (Preliminary  Draft,  2005) 


□aD<aaoD 
m r 

□  aOSMD.Si-PM 

Acquisition 
Modeling  and  Simulation 
Master  Plan 

Preliminary  Draft 

□it* 

Undir  VKFitur  or  DifenM  ter 
AcqmitMm,  T.srsrtnlnjy,  uid  ■.□gnUbi 


■  Establish  policy  to  require  documented  M&S  planning  at  the  joint  capability  & 
program  levels  as  part  of  the  Systems  Engineering  Plan,  T&E  Strategy  and  T&E 
Master  Plan 

■  Establish  policy  on  appropriate  use  of  M&S  to  plan  tests,  to  complement  system 
live  tests,  and  to  evaluate  joint  capabilities 

■  Establish  a  forum  to  clarify  the  characteristics  and  application  of  various 
distributed  simulation  standards  (HLA,  TENA,  DIS,  SI3,  etc.);  examine 
opportunities  for  convergence 

■  Enable  readily-available  distributed  live-virtual-constructive  environments, 
leveraging  related  initiatives 

□  Establish  DoD-wide  standards  for  distributed  environments 

□  Make  candidate  simulations,  labs  and  ranges  compliant  with  these  standards 

■  Centrally  develop  and  maintain  high-priority,  broadly-needed  M&S  tools 


Source:  “Report  of  the  Acquisition  M&S  Working  Group  regarding  an  Acquisition  M&S  Master  Plan,” 
J.  W.  Hollenbach,  2005  Fall  Simulation  Interoperability  Workshop,  Orlando,  FL,  Sep  2005. 


Some  Common  Themes  in  Acquisition  M&S 
Study  Recommendations 


■  That  M&S  can  reduce  T&E  costs  seems  intuitive  and  is  accepted 

■  Need  for  cross-program  M&S  capabilities  (but  how  to  fund  them?) 

■  Need  for  “virtual”  testbeds  (again,  how  to  fund  them?) 

■  Need  for  open  M&S  standards  (applications,  data,  ...) 

■  Concern  for  proprietary  data  /  intellectual  property 

■  Need  for  education  of  acquisition  community  in  M&S 

■  Recommendations  for  pilot  programs  (but  few  ever  funded) 

■  Need  for  incentives  to  use  M&S  (but  few,  if  any,  actionable  ideas) 


A  combination  of  technical,  process, 
and  culture  issues  remain 

- dPL 
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Some  Personal  Observations 
on  the  Way  Ahead 

■  As  cross-cutting  enterprise-level  functional  disciplines  in  the 
acquisition  process,  M&S  and  T&E  have  to  work  together 

□  “If  we  don’t  hang  together,  we  will  surely  ...” 

■  M&S  standards  supporting  T&E  need  to  be  “harmonized”  to  better 
support  multi-Service,  Joint  and  coalition  T&E 

□  HLA  ...  TENA  ...  DIS  ... 

■  A  way  must  be  found  to  fund  “common  good”  M&S-in-T&E  activities 
that  support  multiple  programs  across  Services 

■  A  consensus-based  process  should  be  initiated  to  develop  an  M&S-in- 
T&E  Recommended  Practices  Guide  (RPG) 

□  Use  DMSO’s  DoD  M&S  VV&A  RPG  development  process  as  an  exemplar 

■  Be  cognizant  of  the  recommendations  of  prior  studies,  how  they  were 
received,  and  what  results  they  achieved 

□  “Those  who  ignore  history  are  condemned  to  repeat  it.  ” 


Prepared  Remarks 

Modeling,  Simulation  and  Testing:  Collision  Course  or  Happy 

Marriage? 

Philip  E.  Coyle 
March  7,  2006 

NDIA  22nd  Annual  National  Test  and  Evaluation  Conference 
Hyatt  Regency  Jacksonville 

When  I  chose  as  a  title  for  my  remarks  -  Modeling,  Simulation  and  Testing: 
Collision  Course  or  Happy  Marriage?  - 1  didn't  know  I  would  be  following 
Gen.  Larry  Welch,  always  a  daunting  prospect.  Gen.  Welch  probably  knows 
more  about  modeling  and  simulation  than  any  general  officer  who  ever 
lived.  I  look  forward  to  every  opportunity  I  have  to  hear  him  speak.  I've 
never  known  anyone  who  can  move  from  the  highest  level  policy  issues 
down  to  the  individual  warfighter  with  more  insight  and  clarity.  Gen.  Welch 
moves  with  ease  from  40,000  feet  altitude  down  to  ground  truth,  and 
anywhere  in  between,  and  he  always  makes  great  common  sense,  as  he  has 
again  this  morning.  Mark  Twain  said  "Common  sense  is  not  so  common," 
and  Gen.  Welch  proves  how  true  that  is.  He  also  made  great  common  sense 
in  a  recent  DSB  study,  which  I'll  come  to  in  a  minute. 

When  I  chose  my  title  for  this  talk,  I  also  didn't  know  that  Admiral  Bert 
Johnston  would  be  giving  a  talk  on  Wednesday  titled  "Open  Air  Testing  vs. 
Modeling  and  Simulation:  At  Last  Peaceful  Coexistence?" 

So  before  you  decide  whether  you  like  what  I  have  to  say,  you'd  better  wait 
for  Admiral  Johnston's  talk,  as  well. 

Expectations  for  M&S  -  DOD  vs.  DOE  -  the  differences  and  the  reasons 

But  to  get  back  to  the  marriage  between  modeling,  simulation  and  testing, 
I've  always  been  puzzled  by  the  fact  the  M&S  is  so  problematic  in  the  DOD 
family.  In  the  DOD,  M&S  kind  of  nibbles  around  the  edges,  which  I 
suppose  is  alright  if  that's  all  the  DOD  wants,  but  the  DOD  and  the  Services 
regularly  make  high  sounding  pronouncements  that  M&S  is  going  to  be  the 
answer  and  the  greatest  thing  since  sliced  bread.  While  we  will  hear  during 
this  conference  examples  of  excellent  M&S  work,  it  is  not  easy  to  find 
examples  in  the  DOD  where  M&S  has  really  made  a  difference. 
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Some  of  you  know  that  I  came  to  the  Pentagon  from  the  Lawrence 
Livermore  National  Laboratory  where  modeling,  simulation  and  testing  was 
-  and  still  is  -  a  very  happy  marriage.  We  never  did  a  test  without  first  trying 
to  calculate  -  model,  as  you  would  say  -  in  rather  excruciating  detail  what 
would  happen.  It  was  literally  unthinkable  that  you  would  spend  millions  of 
dollars  on  a  test  without  making  an  equivalent  effort  first  in  M&S. 

For  the  most  part,  the  DOD  does  not  do  that.  It's  quite  common  in  the  DOD 
to  spend  millions  of  dollars  on  a  test  without  making  any  significant 
investment  in  M&S  first.  An  area  of  some  exception  is  Live  Fire  testing, 
about  which  I'll  say  more  later. 

Nevertheless,  I  came  to  the  DOD  fully  prepared  support  M&S,  and  did  so 
throughout  my  tenure  at  the  DOD.  But  the  support  I  gave  to  M&S,  and  the 
support  that  Undersecretaries  for  Defense  for  AT&L  and  many  other  senior 
leaders  have  given  to  M&S,  has  made  very  little  difference.  This  is 
especially  surprising  since  the  computational  tools  that  are  available  today 
are  so  much  more  powerful  than  they  were  ten  or  twenty  years  ago. 
Nevertheless,  DOD  practices  haven't  changed  much.  I  think  there  are  at 
least  four  reasons  for  this. 

First,  the  focus  in  DOD  defense  acquisition  is  on  buying  something  and 
moving  on,  not  on  understanding  for  its  own  sake.  Understanding  is 
important,  of  course,  you  can't  field  a  system  without  understanding  it,  but 
detailed  scientific  and  technical  understanding  -  both  broad  and  deep  -  is  not 
the  first  priority.  By  contrast,  the  culture  in  the  development  of  nuclear 
weapons  has  been  to  achieve  first-principles  understanding  of  everything. 

It's  a  good  thing,  because  today  full-scale  underground  nuclear  testing  is  no 
longer  permissible.  Without  those  models,  the  Department  of  Energy 
weapons  labs  would  be  quite  helpless  today. 

Second,  program  managers  don't  want  to  spend  money  on  M&S  if  they  don't 
have  to.  PMs  have  many  demands  on  their  resources,  and  the  incentives  are 
to  get  the  system  into  production  with  as  little  perturbation  as  possible.  In 
the  Department  of  Energy  (DOE)  family  it  is  assumed  that  you  will  spend 
money  on  M&S,  lots  of  money,  and  it  is  budgeted  every  year,  year  after 
year,  with  no  expectation  that  you  will  ever  be  finished.  Further,  in  the  DOE 
family  there  is  no  expectation  that  you  can  trade  off  M&S  for  testing;  they 
go  together.  So  there  is  no  PM  who  is  trying  to  cut  testing  by  spending 
money  on  M&S. 
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Third,  the  goal  for  modeling  and  simulation  in  the  Department  of  Energy 
family  is  to  be  able  to  predict  with  rather  astonishing  accuracy  what  will 
happen.  Dr.  Keith  Bradley  will  show  some  examples  in  his  talk  Thursday 
afternoon.  While  no  model  is  ever  perfect,  the  goal  is  to  come  as  close  to 
reality  as  possible,  and  often  the  DOE  Labs  get  very  close  to  that  goal.  This 
means  that  modeling  and  simulation,  and  the  evaluations  that  come  from 
those  models,  may  produce  bad  news. 

By  contrast,  in  the  DOD,  the  tendency  is  to  expect  that  test  and  evaluation 
will  produce  bad  news  and  that  M&S  will  produce  good  news.  Thus  M&S 
is  often  recommended  as  the  better  choice,  not  only  because  it  might 
supplant  testing,  but  because  -  generally  -  the  expectation  is  that  M&S  will 
produce  better  news.  Of  course,  this  good  news  may  not  have  anything  to 
do  with  reality,  especially  if  the  models  being  used  were  first  developed  -  as 
is  often  the  case  -  to  sell  the  system,  not  to  understand  it  in  technical  detail. 

In  the  DOE  Labs,  a  model  may  produce  very  bad  news,  bad  news  that  you 
have  to  take  very  seriously  because  the  model  has  been  doing  a  pretty  good 
job  of  predicting  reality.  That  bad  news  may  not  actually  be  bad  news,  as 
subsequent  tests  may  reveal,  but  unless  M&S  and  testing  work  together,  you 
won't  know  for  sure. 

My  point  is  that  in  the  DOE  Labs,  the  intent  is  that  the  models  will  capture 
the  real  world,  just  as  tests  are  trying  to  capture  the  real  world.  Where  does 
a  model  or  simulation  get  its  inputs?  Often  from  test  results.  How  are  the 
test  results  to  be  interpreted  and  understood?  Often  through  modeling  and 
simulation. 

Thus,  in  the  DOE  Labs,  M&S  and  testing  are  partners  that  go  together  with 
more  or  less  equal  likelihood  of  providing  realistic  insights.  To  get  to  the 
bottom  of  the  questions  M&S  and  testing  both  raise,  it  is  necessary  to  iterate, 
back  and  forth,  between  the  models  and  live  tests. 

This  iterative  process  is  essential  for  a  healthy  M&S  program.  Accordingly, 
in  the  DOE  Labs  if  a  model  produced  bad  news,  you  wouldn't  just  change 
the  model  so  it  produced  happier  results,  and  then  stop.  This  is  something 
I've  seen  done  in  the  Department  of  Defense.  If  a  model  produces  bad  news, 
you  shouldn't  just  change  it;  you  need  to  understand  why!  Maybe  the 
problem  is  real.  And  to  find  out  takes  an  iterative  process  that  essentially 


3 


never  ends.  This  type  of  sustained,  iterative  process  between  M&S  and 
testing  is  not  very  common  in  the  DOD. 

Fourth,  it  is  more  difficult  in  the  DOD  to  identify  a  manageable  set  of  M&S 
programs  and  goals.  You  can't  model  everything  -  any  more  than  you  can 
test  everything  -  and  DOD  systems  are  often  so  complex  that  the  number  of 
areas  where  one  could  conduct  M&S  is  mind  numbing. 

In  short,  in  the  DOE  nuclear  weapons  family  there  is  really  only  one  overall 
mission  supported  by  a  relatively  small  number  of  programs,  whereas  in  the 
DOD  family  there  are  many  missions  and  hundreds  of  programs. 

Recommendations  -  Four  Steps  for  Change 

So  to  establish  a  happy  marriage  between  M&S  and  testing  in  the  DOD  it's 
going  to  take  several  new  things: 

First,  DOD  leadership  will  need  to  make  larger  and  sustaining  investments 
in  M&S  that  are  not  tied  to  earlier  full-rate  production  dates  or  reduced 
testing.  DOD  investments  are  needed  in  M&S  for  its  own  sake.  This  doesn't 
mean  that  M&S  won't  have  to  meet  programmatic  goals.  Of  course  it  must. 
But  so  long  as  it  is  a  zero  sum  game,  where  a  dollar  spent  on  M&S  is 
expected  to  save  a  dollar  on  testing,  M&S  and  testing  will  have  no  more 
than  a  shotgun  marriage  relationship,  at  best. 

Second,  M&S  will  have  to  be  supported  by  an  iterative  process  from 
continuing  and  ongoing  testing.  Some  of  these  tests  will  be  conducted 
primarily  to  support  M&S,  and  thereby  understanding,  not  necessarily  to 
support  a  programmatic  milestone.  And  it  also  means  that  the  DOD  will 
sometimes  get  bad  news  from  M&S,  and  when  it  does  it  cannot  react  by 
canceling  the  M&S,  or  cutting  its  budget,  or  cutting  its  reach. 

Third,  to  make  these  investments  manageable,  the  DOD  will  need  to  focus 
on  mission  areas,  not  programs.  Here  is  where  the  DSB  report  that  Gen. 
Welch  co-chaired  with  Bob  Herman  is  important.  That  report  -  the  DSB 
Summer  Study  on  Transformation  -  points  out  that  the  DOD  must  discipline 
its  resources  to  mission  purposes.  Under  the  current  approach,  DOD  always 
has  many  more  programs  all  fighting  for  dollars  than  it  can  afford  -  the  bow 
wave  effect  in  "a  sea  of  force  provider  interests".  The  DSB  Summer  Study 
recommends  -  among  many  other  things  -  allocating  resources  to  mission 
purposes,  constraining  plans  to  intended  resources,  and  measuring  progress 
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against  plan  objectives.  The  DSB  study  also  recommends  -  for  the  reasons 
Gen.  Welch  explained  a  few  minutes  ago  -  limiting  "the  first  stage  of  spiral 
development  to  designs  providing:  A  useful  increment  of  added  military 
capability  where  there  is  no  more  than  moderate  risk  in  achieving  cost, 
schedule  and  performance  goals." 

Measuring  progress  against  plan  objectives,  and  measuring  risk  requires 
modeling,  simulation,  test  and  evaluation,  all  working  together. 

In  this  regard,  Live  Fire  testing  is  an  area  where  M&S  practices  have  been 
laudable.  I  don't  know  how  DOT&E  does  it  these  days,  but  Jim  O'Bryon 
used  to  insist  that  a  Live  Fire  test  not  be  conducted  until  it  had  been  modeled 
first.  I  thought  this  was  a  very  important  policy  and  supported  him  in  it 
throughout.  He  didn't  always  get  what  he  wanted.  Sometimes  the  M&S 
work  was  poor  or  half-hearted,  sometimes  the  Services  just  wanted  to  do  the 
test  and  get  it  over  with,  but  Jim  tried  to  instill  in  Live  Fire  testing  a 
partnership  with  M&S,  a  relatively  happy  marriage. 

In  this  sense  you  could  think  of  Live  Fire  testing  as  a  kind  of  "mission"  area. 
Metal  hitting  metal  is  something  that  has  to  be  expected  with  most 
battlefield  systems,  and  so  you  could  think  of  Live  Fire  testing  as  something 
common  and  overarching  individual  programs.  Often  the  details  in  Live 
Fire  tests  might  be  quite  common  between  programs  that  were  otherwise 
quite  different.  It  is  in  this  sense  that  one  can  think  of  Live  Fire  testing  as  a 
"mission"  area  not  unlike  the  context  in  which  the  word  "mission"  is  used  in 
the  DSB  report,  although  they  are  referring  to  mission  areas  in  an 
operational  warfighting  context.  And  in  this  sense  one  can  imagine 
organizing  and  managing  M&S  efforts  along  mission  lines,  as  well. 

My  only  point  here  is  that  by  focusing  on  mission  areas  -  not  necessarily 
individual  procurements  -  just  as  the  DSB  report  recommends,  M&S  also 
could  better  manage  the  dazzling  array  of  tasks  it  might  undertake. 

The  emphasis  in  the  DSB  study  on  increased  oversight  and  measuring  risk, 
brings  me  to  a  fourth  needed  step.  To  adequately  measure  risks,  the  DOD 
will  need  to  improve  its  understanding  -  broad  and  deep  -  of  the  mission  area 
systems  it  intends  to  field.  This  new  emphasis  on  understanding  -  so  as  to 
truly  understand  the  risks  involved  -  will  be  good  for  M&S  also,  and  will 
produce  a  climate  where  understanding  -  not  just  procurement  -  is  valued  for 
its  own  sake. 
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The  Defense  Acquisition  Performance  Assessment  Report 

Having  mentioned  how  helpful  I  believe  Gen.  Welch's  DSB  report  on 
Transformation  can  be  -  not  only  to  our  overall  military  establishment,  but 
by  extension  to  how  we  think  about  M&S  in  the  DOD,  I  need  to  comment 
on  the  Defense  Acquisition  Performance  Assessment  Report,  which  came 
out  last  month.  The  DAPA  report,  so  called,  complete  with  glossy 
cardboard  cover  and  a  set  of  cardboard  "3-D"  prism  glasses  such  as  seen  in 
science  fiction  movies  in  the  1950s,  is  neither  effective  nor  suitable.  What 
the  "3-D"  glasses  are  for  is  a  mystery,  the  DAPA  links  to  their  web  sites 
don't  work,  and  the  little  prisms  at  the  bottom  of  each  page  don't  shed  any 
light  on  the  real  problem. 

In  fairness,  the  DAPA  report  points  out  that  over  the  years  there  have  been 
128  separate  studies  of  DOD  acquisition  performance,  this  being  the  129th  - 
and  does  quote  others  to  identify  certain  problems.  On  its  front  page,  the 
DAPA  report  quotes  two  recent  Acquisition  Policy  reports  by  the  Congress. 
The  first  from  the  Senate  says,"  Problems  occur  because  the  Department  of 
Defense's  weapons  programs  do  not  capture  early  on  the  requisite 
knowledge  that  is  needed  to  efficiently  and  effectively  manage  program 
risks." 

The  second  quote  from  the  House  says,  "...that  the  current  Defense 
Acquisition  Framework  is  not  appropriately  developing  realistic  and 
achievable  requirements  within  integrated  architectures  for  major  weapons 
systems  based  on  current  technology,  forecasted  schedules,  and  available 
funding..." 

The  Congress  clearly  understands  the  problem.  So  does  the  GAO  who 
reported  way  back  in  1971  that  "untested  and  undetermined  technology 
risks"  and  "poorly  defined  requirements"  were  two  major  factors  why  DOD 
acquisitions  so  often  failed.  Of  course  the  GAO  has  pointed  this  out 
countless  times  since  1971,  also. 

However,  the  DAPA  report  does  NOT  own  up  to  the  principal  problems 
already  well  identified  by  the  Congress  or  the  GAO. 

While  the  DAPA  report  makes  1 ,069  recommendations  in  42  issue  areas, 
most  prominently,  the  need  for  improved  oversight,  it  never  addresses  the 
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problems  with  risk  that  the  Congress,  the  GAO,  and  Gen.  Welch's  DSB 
report  address  so  clearly. 

How  can  this  be,  you  ask?  In  this  issue  area  -  the  need  for  improved 
oversight  -  the  DAPA  report  made  more  recommendations  than  in  any  other 
issue  area.  Yes,  but  instead  of  recommending  mechanisms  to  strengthen 
oversight  and  the  understanding  of  risk  -  such  as  earlier  and  more  rigorous 
testing,  and  concomitant  efforts  in  M&S,  -  the  DAPA  report  recommends 
constraining  oversight,  and  constraining  the  scope  of  OT&E,  operational  test 
and  evaluation,  which  is  hardly  the  problem  when  it  comes  to  understanding 
risk. 

If  in  the  DOD,  M&S  actually  identified  risks,  as  I  am  recommending,  the 
DAPA  report  probably  would  have  recommended  constraining  M&S  as 
well.  Thus  the  DAPA  report,  if  implemented,  would  be  bad  for  the  future  of 
M&S  and  the  larger  role  M&S  could  play. 

In  addition  to  being  silent  on  addressing  risk  and  technological  maturity,  the 
DAPA  report  does  little  to  address  the  incentives  needed  to  change  behavior. 
You  all  know  that  incentives  are  what  drive  any  process,  and  a  lack  of 
incentives  is  part  of  the  reason  why  modeling  and  simulation  is  so  different 
in  the  Department  of  Defense  than  in  the  Department  of  Energy. 

Also,  the  DAPA  report  does  not  address  the  many  problems  in  contracting 
that  come  from  an  inability  of  the  government  to  get  needed  information. 

This  latter  point  often  comes  up  in  an  M&S  context  where  contractor  models 
-  originally  developed  to  sell  a  system  to  Congress  and  to  senior  DOD  and 
Service  leadership  -  are  treated  as  proprietary  under  many  contracts  even 
though  those  models  are  then  used,  sometimes  misused,  in  the  development 
of  a  system. 

So  unless  you  want  to  give  the  3-D  glasses  to  your  kids,  I  wouldn't  waste 
your  time  on  the  DAPA  report. 

M&S  and  Testing  -  Sharing  the  Same  Goals 

Jack  Krings  has  spoken  candidly  about  these  things  at  many  conferences. 
And  Tom  Christie  has  spoken  and  written  about  them  as  well.  In  the 
February,  2006,  issue  of  Naval  Institute  Proceedings  Magazine,  Tom 
Christie  has  published  a  wonderful  article  titled,  What  Has  35  Years  of 
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Acquisition  Reform  Accomplished?  If  you  haven't  seen  his  paper,  I 
recommend  it.  As  Tom  explains  so  well,  in  system  after  system  -  time  after 
time  -  the  front  end  of  the  acquisition  process  sows  the  seeds  of  future 
problems.  To  quote  Tom,  ". . .by  the  time  technical  and  cost  issues  come  to 
the  fore  in  spades,  few,  if  any,  of  those  involved  in  the  process  can  bring 
themselves  to  admit  they  were  wrong,  to  cut  their  losses  before  inevitable 
further  cost  growth  and  schedule  slips,  or  to  demonstrate  much  needed 
discipline  by  making  an  example  of  program  officials  and  their  contractors 
who  have  sold  the  department  and  the  taxpayer  a  bill  of  goods." 

Tom  points  out  that  what  is  needed  is  discipline,  exactly  what  Gen.  Welch's 
DSB  report  on  transformation  also  recommends.  And  modeling  and 
simulation  can  be  part  of  that  discipline  in  the  DOD,  just  as  it  is  in  the  DOE. 

Like  all  successful  marriages,  for  modeling,  simulation  and  testing  to  be  a 
happy  marriage  they  have  to  share  the  same  goals  -  namely  to  understand 
how  a  system  really  works,  its  strengths  and  weaknesses,  and  whether  the 
system  will  be  effective  and  suitable  for  the  warfighter.  The  purpose  of 
M&S  and  testing  together  is  to  produce  insight  and  understanding.  You  all 
know  from  your  own  work  that  when  you  can  measure  a  result  in  a  test,  and 
then  calculate  -  model  -  that  same  result  from  first  principles,  that's  when 
you  feel  you  really  understand  it.  You  may  still  have  many  questions,  but 
you  know  you  are  better  off  than  you  would  be  from  either  testing  or  M&S 
alone. 

Conclusion 

The  future  of  M&S  in  the  Department  of  Defense  will  be  determined  by  the 
goals  that  are  set  for  M&S.  The  relationship  between  modeling,  simulation, 
testing  and  evaluation  will  be  determined  by  those  goals  also.  With  the 
computational  tools  that  are  available  today,  there  is  no  longer  an  excuse  for 
the  DOD  to  expect  so  little  from  M&S. 

If  M&S  is  going  to  continue  to  only  nibble  around  the  edges,  if  M&S  is 
going  to  be  viewed  as  a  source  for  good  news  and  never  for  bad  news,  and  if 
M&S  is  thought  of  as  a  way  to  cut  or  avoid  testing,  then  peaceful 
coexistence  may  be  the  best  we  can  hope  for.  For  M&S  to  not  just  nibble 
around  the  edges,  transforming  leadership  is  needed  as  Gen.  Welch  calls  for. 

But  if  M&S  adopts  the  goals  of  true  understanding,  and  providing 
evaluations  that  both  require  and  complement  testing,  then  a  happy  marriage 
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will  result.  To  make  a  difference  to  the  warfighter,  both  M&S  and  testing 
have  to  deal  with  the  real  world.  Testing  already  does  that.  It's  time  that 
M&S  did  too. 

Thank  you. 


9 
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Summary  of  Questions 

mmTMV*i*xtvt m  nnMUisr 

Question  Category 
(becomes  an  Issue) 

#  of  Questions 
that  fall  into 
Category 

Potential  Activity  to 
Address  Issue 

Priority 

Actionee 

Acquisition 

(Potential  issue  to  address 
from  these  questions  is 
development  of  a  Test 
Strategy  White  Paper  for 
Programs  to  leverage) 

6 

System  vs.  Capability  Focus. 

Industry  involvement  in  live- 
virtual  system  development. 

Difference  between  M&S  Mast 
Plan  and  Sim  Supp  Plan. 

Motivation  for  PO  to  perform 
joint  events  in  SDD  with  risk 
from  uncertain  models? 

(Funding  for  Multi-Service  events?) 

Can  DoD  streamline 
acquisition  process  using  M&S 
as  enabler? 

Definitions 

1 

M&S  Savings  Metrics. 
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Summary  of  Questions 
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Question  Category 
(becomes  an  Issue) 

#  of  Questions 
that  fall  into 
Category 

Potential  Activity  to 
Address  Issue 

Priority 

Actionee 

Re-use 

2 

Manage  re-use  of  DT&E  and 
M&S  assets  through  program 
life-cycle  and/or  across  DoD 
Programs. 

Partnerships 

3 

List  of  avail  M&S. 

Confidence  of  sims  that  model 
sub-systems  of  “other” 
contractors  (Intellectual  Property?) 

National/Intemational  Society 
supporting  M&S 
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Introduction 
Mr.  Di Petto 


Ktu^crra  rHim  -i  tm*  >rtt  a  tipi  use* 


•  Testers  need  to  assist  PMs  to  develop  M&S 
strategies 

•  Ensure  Accreditation  in  addition  to  V&V 

•  M&S  must  support  T&E  and  SE 

•  M&S  is  only  practical  means  of  SoS  capability 
testing 

•  Use  Model  -  Test  -  Fix  -  Model  approach 

•  Many  cross-cutting  issues  exist  that  are  beyond 
the  reach  of  PMs  &  testers;  OSD  is  addressing 


Dr.  Kai’liwai 
AFFTC 


FLtttM.ti  R ikOITA  tMrl  vm  A  T1  ilNfl 


•  How  does  DT&E  use  M&S  data  in  testing  today? 

-  Is  M&S  used  to  help  determine  how  much  testing  is 
enough? 

-  Are  M&S  data  used  to  improve  DT&E? 

•  How  does  the  DT&E  community  feed  test  data  to 
M&S  community,  and  vice  versa? 

-  Are  test  data  used  to  refine  M&S  Models? 

•  How  is  DT/M&S  community  working  together  in 
NCO/W? 

-  Are  there  lessons  learned  that  can  apply  to  DT&E/M&S 
communities  elsewhere? 


Mr.  Berard 
WSMR 


FLtttM.ti  li'UK  -i  JH  tMrl  ytrt  A  1H1MUICV 


•  Data  Management 

-  Cross  Domain  (class/unclassified)  solutions 

-  Access/Distribution  Across  the  Distributed  Domain 

•  Integrated  Joint  Processes  (Data,  Architecture, 
Models,  Control,  Scheduling) 

•  Enforcement  of  DoD  standards  and  directives 

-  Joint  Testing  Policy 

•  Configuration  Management 

-  Document  Management,  Networks,  Applications, 
Terrain 

•  Persistent  Multi-Level  Secure  Networks 


Mr.  Cameron 
Boeing 
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•  M&S  is  critical  for  successful  DT&E  of  complex,  net-enabled 
Systems  of  Systems 

-  Requires  robust  application  of  M&S  across  the  entire  product  life 
cycle 

•  M&S  is  available,  today,  to  provide  credible  DT&E  involving 
Live,  Virtual,  and  Constructive  elements 

•  Partnerships  are  key  -  both  government/industry,  and 
industry/industry 

•  Need  to  effectively  deal  with  impediments 

-  Conflicting  programmatic/DT&E  goals 

-  Intellectual  Property 

-  Security 

-  M&S  “zealotry” 


Mr.  Hazlett 
Raytheon 


Ktu^crra  rhbxrjq  mu  vm  i  nnMUisr 


•  Develop  plans  to  better  integrate  M&S  into  design  and  T&E  of  future 
forces 

•  Create  a  opportunity  to  facilitate  T&E  and  M&S  exchange  and 
interactions 

-  Include  specific  topics  such  as  Electronic  Attack  and  C4ISR 

•  Integrate  M&S  into  development  and  acquisition  programs  in  a  value- 
added  manner 

•  Integrate  M&S  into  development  and  acquisition  programs  in  a  value- 
added  manner  to  facilitate  integrated  development  and  T&E 

•  Streamline  and  update  DoD  testing  security  protocol  to  keep  current 
with  changing  requirements  to  protect  developmental  capabilities 

•  Think  about  bringing  in  the  next  generation  individuals 
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Common  Themes 


hYi!1,:;1!,  r.  i-r.Ni 


stUM-ti  li'uic  -i  jh  tMri  vrrr  a  tthmucy 


•  Acquisition 

-  Sharing/Use  of  data  from  M&S  to  T&E  and  from  T&E  to  M&S 

-  Mapping  of  M&S  Capabilities  to  T&E  Requirements 

-  Model  -  Test  -  Fix  -  Model  approach 

•  Definitions 

-  Develop  a  common  terminology  across  M&S  and  T&E 


Common  Themes 

MiMl 

IWII% 

ruioi  c m  «  tijjwh.  c*,\ 

•  Intellectual  Property 

-  How  to  share  and  control  data/models? 

-  Connectivity  concerns? 

•  Security 

-  How  to  deal  with  secure  networks  across  government/industry 
boundaries? 

•  Partnerships 

-  Is  there  a  body  required  to  foster  technical  interchange  of  ideas  and 
develop  proposals  for  standards? 
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Summary  of  Questions 


1*1 

Miriam 

sikwti  nikoiTA  tMri  vrrr  a  tthmucy 


Question:  What  is  the  difference,  if  any,  between  the  M&S  Master 

Plan  versus  the  Simulation  Support  Plan?  Isn't  the  SSP 
meant  to  be  the  "M&S  Master  Plan"  for  a  program? 

Respondent:  Chris  DiPetto 


Answer:  The  Acquisition  M&S  Master  Plan  is  a  broad  over-arching 

plan,  developed  by  representatives  of  all  DoD  components. 
It  identifies  obstacles  which  hinder  effective  and  efficient 
use  of  M&S  in  acquisition,  and  lays  out  a  set  of  27  specific 
actions  to  address  those  obstacles.  Simulation  Support 
Plans  are  used  by  some  program  offices  to  document 
specific  plans  for  using  M&S  within  their  particular  program. 
Implementing  the  actions  in  the  Acquisition  M&S  Master 
Plan  should  facilitate  better  M&S  planning  and  execution  by 
programs. 
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Summary  of  Questions 


1*1 

Miriam 

sikwti  nikoiTA  tMri  vrrr  a  tthmucy 


Question:  Is  there  any  plan  to  establish  reportable  metrics  on  how 

M&S  effort  saved  resources  in  time,  dollars,  and  system 
design? 

Respondent:  George  Kai'liwai 


Answer:  If  there  aren't  plans  to  use  reportable  MS  metrics,  there 

should  be.  In  developing  MS  capabilities  we  assume  a 
certain  ROI  and  generally  get  agreement  from  the  program 
offices  that  they  would  use  these  MS  capabilities  as  long  as 
they  don't  have  to  pay  for  them.  Whether  they  use  the  MS 
capabilities  once  we  deliver  them  is  a  different  matter 
altogether  as  the  TE  community  cannot  force  these 
capabilities  on  our  customers. 
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Summary  of  Questions 


1*1 

Miriam 

sikwti  nikoiTA  tMri  vrrr  a  tthmucy 


Question:  How  can  DoD  streamline  the  acquisition  process  using 

M&S  as  an  enabler,  e.g.  what  architecture,  standards,  and 
tools  can/could  accomplish  this? 

Respondent:  George  Kai'liwai 

Answer:  We  are  embarking  on  an  effort  to  define  the  Value  Stream 

for  our  operations.  This  will  help  us  to  stream-line  our 
operations,  and  improve  upon  the  value  we  provide  to  our 
customers.  Another  way  to  potentially  streamline  the 
acquisition  process  is  to  deploy  systems  for  evaluation, 
rather  than  embark  on  a  operational  test  event.  Obtain  the 
operations  test  information  from  direct  use  in  a  conflict. 
One  way  to  streamline  testing  is  to  deploy  prototypes  into 
the  field  and  if  possible  conduct  DT  and  OT  in  the  field. 
This  would  require  close  coordination  with  the  field 
commander. 
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Summary  of  Questions 


1*1 

Miriam 

sikwti  nikoiTA  tMri  vrrr  a  tthmucy 


Question:  What  M&S  capability  will  be  "left  behind"  (reuseable)  after 

the  April  event  with  Boeing  &  DMOC?  (planning  process, 
network  connectivity,  models,  etc.) 


Respondent:  George  Kai'liwai 


Answer:  Technical  expertise  is  the  primary  aspect  that  we  will  walk 

away  with.  Particularly  expertise  in  distributed  testing. 
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Summary  of  Questions 


.tn  Tiymri  i« 

Miriam 

Fi,tUM.ra  rhkoir.fi  mu  vrrr  a  mmauat 


Question:  Where  can  a  PM  go  today  to  find  a  lit  of  all  the  M&S  currently 

in  use  within  DoD,  to  see  what  might  be  of  use? 

Respondent:  Jim  Hazlett 

Answer:  PMs  should  check  with  DMSO,  and  their  service's  modeling  and 

simulation  organizations,  such  as  AMSO  and  NMSO.  This  should 
catch  most  of  the  M&S  being  used  in  support  of  acquisition 
programs  and  programs  of  record.  As  a  double  check,  one  should 
also  run  the  list  provided  by  the  service  MSO  by  the  system 
commands  (i.e.,  NAVSEA,  NAVAIR,  etc.).  This  should  also  catch 
the  M&S  being  used  at  Warfare  Centers.  Finally,  there  is  M&S  that 
is  being  used  by  DoD  that  is  not  necessarily  program-related 
(wargaming  models,  etc.).  To  catch  this  one  should  check  with  the 
war  gaming  departments  at  the  service  colleges,  and  at  the  doctrine 
commands  and  battle  centers.  MORS  and  several  other 
professional  organizations  do  keep  some  databases  on  DoD  M&S 
use. 
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Summary  of  Questions 


1*1 

Miriam 

sikwti  nikoiTA  tMri  vrrr  a  tthmucy 


Question:  Why  establish  a  DoD  society  and  not  a  National 

(International?)  organization?  Probably  more  potential 
members  outside  DoD  than  within? 


Respondent:  Jim  Hazlett 

Answer:  Agree  that  a  modeling  and  simulation  society  should  be  more 

inclusive  than  just  DoD,  but  it  should  also  be  organized  to 
include  functionally-focused  components  (service,  DT&E, 
OT&E,  acquisition,  open  source,  gaming,  etc.)  There  is  a 
strong  argument  to  make  this  a  "virtual"  organization,  due  to 
the  nature  and  diversity  of  the  subject  area.  A  strong  web- 
based  effort  would  probably  be  the  best  way  to  get  this  going. 
There  could  be  both  national  and  international  branches, 
where  one  could  matrix  themselves  in,  as  appropriate.  The 
national-international  linkage  should  be  such  that  it  accounts 
for  export  control  issues,  etc. 
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Summary  of  Questions 


1*1 

Miriam 

sikwti  nikoiTA  tMri  vrrr  a  tthmucy 


Question:  What  can  be  done  (given  need  for  flexibility  in  DT&E  and 

rigorous  VV&A  for  OT&E  and  other  issues)  to  facilitate  re-use 
of  DT&E  assets  through  the  remainder  of  the  program  life- 
cycle? 

Respondent:  Jim  Hazlett 


Answer:  As  spiral  development  continues  to  become  the  more 

common  way  of  development,  acquisition,  and  testing,  the 
line  between  D  and  O  T&E,  and  M&S  will  blur.  We  are  now 
facing  rapidly  evolving  threats  (i.e.,  Improvised  Explosive 
Devices  (lEDs)  and  asymmetric  warfare)  that  will  require  a 
blend  of  DT&E,  DM&S,  and  OT&E  and  OM&S.  We  will  see 
"build  a  little,  model  a  little,  test  a  little,  field  a  little  (or  a 
lot)... and  repeat."  FCS  and  other  future  programs  are  likely  to 
evolve  steadily  over  their  lifespan,  and  should  be  designed  to 
accommodate  this  reality. 
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Summary  of  Questions 

FLtttV.ti  nitcm .H  Mt  Itn  A  T1  ilWRffjV 

Question: 

There  is  much  DoD  talk  about  distributed  environments  for  systems 
engineering  and  test.  How  can  we  bring  industry  into  the  planning, 
so  in  the  future  industry  will  be  able  to  propose  and  use  persistent 
live-virtual-constructive  environments  in  systems  development? 

Respondent: 

Steve  Cameron 

Answer: 

The  very  best  thing  to  do  would  be  to  ensure  industry  and  government  are 
involved  together  from  the  very  beginning.  We  have  learned  in  the  past  that 
having  the  government  develop  an  environment,  then  tell  industry  to  use  it 
doesn't  work  well.  It  is  important  to  treat  both  industry  and  government  as 
partners  during  the  requirements  stage  for  such  an  environment  as  well  as 
during  its  actual  development.  Industry  specific  issues,  such  as  the 
protection  of  Intellectual  Property,  need  to  be  treated  as  a  concern  for  all 
team  members  (government  and  industry),  and  treated  up  front  with  the  rest 
of  the  design  requirements.  In  JMASS,  we  saw  that  the  government  can't  do 
it  on  its  own,  and  in  ACD&D  we  learned  industry  can't  do  it  on  its  own.  The 
answer  is  "combined  (industry/government)  development  from  requirements 
to  deployment." 
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Summary  of  Questions 


1*1 

JiTMF^lK  RliDST*  ItSIrl  yttt  i  lUIMUISr 


Question:  You  said  that  "LabNet  enables  ad  hoc  simulation  networking." 

Why  "ad  hoc"  vice  "persistent?" 


Respondent:  Steve  Cameron 


Answer:  I  used  the  term  “ad  hoc”  to  mean  a  particular  user  can 

configure  LabNet  however  needed  for  a  particular  use.  This 
ability  allows  us  to  conduct  distributed  M&S  to  virtually  any 
security  level.  This  particular  configuration  can  then  remain 
continuously  available  (persistent)  for  as  long  as  the  user 
requires. 
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Summary  of  Questions 


1*1 

itnncn  nikoiTA  mk  vrrr  t  nnMUisr 


Question:  What  motivates  a  System  PO  to  conduct  a  Joint  distributed 

event  during  System  Development  to  find  bad  news 
(potentially)  with  M&S  tools  that  are  uncertain  and  networks 
that  are  difficult  to  establish? 

Respondent:  Steve  Cameron 

Answer:  This  shouldn’t  become  an  issue  of  forcing  people  to  use 

M&S.  Program  Managers  will  use  M&S  when  it  helps  them 
best  balance/manage  cost,  schedule,  and  performance. 
Instead,  we  should  be  focusing  on  helping  a  Program 
Manager  determine  when  and  how  to  use  M&S.  One  glaring 
hole  in  this  capability,  is  that  we  have  never  really  been  able 
to  truly  show  the  worth  of  M&S  (in  advance  of  its  use... so  its 
use  can  be  weighed  from  a  cost/benefit  perspective). 
Something  perhaps  easier  to  do  is  to  create  a  better  decision 
template  for  M&S,  such  as  the  Acquisition  M&S  Master  Plan 
is  suggesting. 
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Summary  of  Questions 


.tn  Tiymri  i« 

Minim. 

™^lll  DlkfKTih  JUrl  yttt  A  ftlMMtCXA 


Question:  In  your  Virtual  WarFare  Center,  your  slide  shows  it  can 

simulate  AEGIS  system.  How  do  you  validate  the  AEGIC 
model  (which  Lockheed  Martin  maintains),  or  are  you 
partnering  with  them?  For  example,  if  it  is  not  a  Boeing 
product,  what  is  the  confidence  of  non-Boeing  product 
simulations  in  your  Virtual  Laboratory? 

Respondent:  Steve  Cameron 


Answer:  This  is  a  Lockheed  Martin  model  used  on  a  program  with 

which  Lockheed  Martin  was  a  partner,  and  is  used  when  the 
agreement  for  its  use  applies.  In  the  general  sense,  industry 
obtains  validation  of  models  to  the  satisfaction  of  the 
customer  for  that  simulation.  For  systems  in  our  simulations 
which  we  do  not  produce,  our  company  attempts  to  use 
models  that  have  been  validated  by  the  Customer. 
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Summary  of  Questions 


1*1 

JiTMF^lK  RliDST*  ItSIrl  yttt  i  lUIMUISr 


Question:  I  note  you  mentioned  that  an  impediment  to  testing  was  the 

inability  to  achieve  statistical  significance.  Do  you  really 
mean  this? 


Respondent:  Tom  Berard 

Answer:  Yes.  A  data  point  of  one  is  hardly  "statistically  significant." 

I'd  call  that  a  "demonstration."  That  does  not  mean  that 
there  isn't  a  significant  amount  of  information  gained  from 
such  a  demonstration.  It's  a  great  learning  experience.  I 
also  believe  that  if  we  have  done  our  job  correctly  and  used 
M&S  to  characterize  the  test  event,  it  can  be  used  to 
validate  the  models  used. 
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Summary  of  Questions 


.tn  Tivmri  \km 

Minim. 

IK  DlkOITA  Mrl  A  Ti  iTWFLffrjY 


Question:  Who  funded  the  multi-service  distributed  event?  Can  you 

speak  to  the  coordination  required  to  make  it  happen? 


Respondent:  Tom  Berard 

Answer:  Air  Force,  Navy,  Army  paid  for  their  service  contributions. 

JFCOM  was  not  a  bill  payer  for  this  event. 
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Session  2 
March  7,  2006 


-  DT&E’s  Perspectives  - 
Making  M&S  and  T&E  Better  Partners 


Panel  Chair:  Chris  DiPetto 

Special  Assistant,  Systems  Engineering 

Office  of  the  Under  Secretary  of  Defense 
for  Acquisition,  Technology,  and  Logistics 


NDIA  T&E  Conference 
Session  2,  March  7  2006 


IgP  -  DT&E’s  Perspectives  - 

“Making  M&S  and  T&E  Better  Partners” 

Panel: 

*  Dr  George  Ka’iliwai 

Technical  Advisor,  USAF  Flight  Test  Center,  Edwards  AFB 

*  MrTomBerard 

Director,  USA  White  Sands  Missile  Range 

*  Mr  Steve  Cameron 

Sr  Manager,  M&S  Infrastructure,  The  Boeing  Corporation 

*  Mr  James  A.  Hazlett 

Director,  Mission  Profiling,  Raytheon  Corporation 


NDIA  T&E  Conference 
Session  2,  March  7  2006 


Acquisition  Challenges  Today 

•  Capabilities 

-  Warfighters  want  better  capability,  now,  versus  ideal 
capability,  later 

-  Capability-acquisition  management  versus  platform- 
acquisition  management 

•  Mission  &  System  Complexity 

-  Broad  spectrum  of  missions;  Joint  and  Network-Centric 
Operations 

-  Many  cross-platform  interactions  and  dependencies;  IT 
intensive  systems 

•  USD(AT&L)  imperative: 

“Help  drive  good  systems  engineering  practices 
back  into  the  way  we  do  business.” 


NDIA  T&E  Conference 
Session  2,  March  7  2006 


Relevant  Acquisition  Policy 

DoDI  5000.2  May  12,  2003 


E5.1  The  PM. ..shall  coordinate  DT&E,  OT&E, 
LFT&E,  family-of-systems 
interoperability  testing,  information 
assurance  testing,  and  M&S  activities, 
into  an  efficient  continuum,  closely 
integrated  with  requirements  definition 
and  systems  design  and  development. 


E5.4.7  Appropriate  use  of  accredited  models 
and  simulation  shall  support  DT&E, 
IOT&E,  and  LFT&E. 


NDIA  T&E  Conference 
Session  2,  March  7  2006 


Defense  Acquisition  Guidebook 

9.3.4.Modeling  and  Simulation  in  DT&E 

•  M&S  is  integral  to  and  inseparable  from  T&E 

-  Test  to  confirm  system  performance  &  validate  M&S 

-  Better  understanding  of  interaction  of  system  & 
environment  than  with  either  M&S  or  testing,  alone 

•  PM’s  M&S  WIPT  develop  M&S  strategy;  T&E 
manager  assist 

-  Plan  and  start  M&S  early  in  program;  evolve  utility 
(update)  across  program's  life 

-  System  M&S  used  in  system  test  should  be  the  same  as, 
or  traceable  to,  M&S  used  for  concept  development, 
analysis-of-alternatives,  system  design,  and  production 

-  Synthetic  test  environments  may  also  be  reused  for 
training,  operations  planning  and  rehearsal,  and 
subsequent  concept  developments 

NDIA  T&E  Conference 
Session  2,  March  7  2006 
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Issues  facing  M&S  in  Test 

*  Planning  program  life  cycle  strategy  for  M&S 

-  Knowledgeable  staff;  experienced,  skilled  help 

-  Integrated  approach,  with  industry  prime 

-  Leveraging  reuse  opportunities  (despite  contractor  proprietary 
concerns  and  self-interest) 

-  Contract  language,  data  rights 

*  Considering  and  getting  needed  representations  from 
others 

-  Discovering  and  understanding  other  representations 

-  Access  problems  of  proprietary  considerations,  misuse  concerns 

-  Compensation  to  representation  providers 

*  W&A 

*  Security  accreditation  of  distributed  environments 

*  Improving  M&S  with  empirical  data,  when  no  one  has  $ 

*  Maintaining  and  improving  broadly-needed  M&S 

^  1  NDI/mE  Conference  J 

Session  2,  March  7  2006 


6 


Resolving  the  M&S  Issues 

*  27  actions  in  draft  Acquisition  M&S  Master  Plan 

-  Five  goals 

•  Provide  necessary  policy  and  guidance 

•  Enhance  the  technical  framework  for  M&S 

•  Improve  model  and  simulation  capabilities 

•  Improve  model  and  simulation  use 

•  Shape  the  workforce 

-  Details:  see  Fred  Myers’  paper,  Thursday  March  9, 1PM 

-  1st  step;  a  long  way  to  go 

•  Acquisition  M&S  Working  Group 

-  Voice  of  the  Acquisition  M&S  community 

-  Review  and  assess  execution  of  Master  Plan’s  27 
actions 

-  Identify  &  submit  needs  for  central  funding 

NDIA  T&E  Conference 
Session  2,  March  7  2006 


Summary 

Making  M&S  and  DT&E  Better  Partners 


•  Testers  need  to  assist  PMs  to  develop  M&S 
strategies 

•  M&S  must  support  testing  and  SE 

•  M&S  is  only  practical  means  of  SoS 
capability  testing 

•  Many  cross-cutting  issues  exist  that  are 
beyond  the  reach  of  PMs  &  testers;  OSD  is 
addressing 


NDIA  T&E  Conference 
Session  2,  March  7  2006 
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Live-  Virtual-Constructive 
Distributed  Environments 


nuatJSational  Test  and  Evaluation  Conference 


in  Support  of  Test  and  Ev 


A.  Greene,  PhJ). 

Air  Force  Agency  for  MoneU^jsiottiioUoo 


U.S.AIR  FORCE 


Outline 

LVC  Distributed  Environments  in  Support  of  T&E 


■  Air  Force  Agency  for  Modeling  and  Simulation  (AFAMS)  Mission 

■  Air  Force  Modeling  and  Simulation  (M&S)  Thrusts 

■  Accelerated  Acquisition 

■  Live-Virtual-Constructive  (LVC)  Distributed  Environments 

■  Air  Force  Integrated  Collaborative  Environment  (AF-ICE) 

■  Joint  Mission  Environment  Test  Capability  (JMETC) 

■  Test  and  Evaluation  Activities  using  LVC  Distributed  Environments 

■  Multi-Service  Distributed  Event  (MSDE) 

■  Joint  Expeditionary  Force  Experiment  (JEFX) 

■  Joint  Test  and  Evaluation  (JT&E)  /  Joint  Feasibility  Study  (JFS) 
Activities  using  LVC  Distributed  Environments 

■  Joint  Test  Environment  Methodology  (JTEM) 

■  Joint  Command  and  Control  of  Net-Enabled  Weapons  (JC2NEW) 

■  Our  Future 

■  Net-Centricity  and  Ultimately  Net-Enabled  Decision-Centricity 

■  Testing  in  a  Joint  Environment 

■  Key  Pieces  to  Success 


Integrity  -  Service  -  Excellence 
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U.S.AIR  FORCE 


orce  Modeling 


Air  Force  Agency  for  Modeling  and  Simulation,  Orlando  Florida 


Role 

Air  Force  lead  for  translating  policy/direction  into  effective  M&S  capabilities  integrated  across 
the  Air  Force  and  with  the  rest  of  the  Department  of  Defense 


Chain  of  Command 

■  SAF 

■  SAF/XC 

■  SAF/XCO 

■  AFAMS 

People 

■  74  Total 

■  19  Military 

■  16  Civilians 

■  39  Contractors 

Liaisons 

■  AFRC  ■  DMSO 

■  AFRL  ■  SOCOM 

■  AF/XO 


Laying  the  Foundation 

•  M&S  Foundations 

•  Databases/Repositories 

•  Standards/Coord  Tools 

•  Architectures/Reqs 

•  AF  M&S  Strategy 

Warfighter  Applications 

•  Trng/Msn  Rehearsal/Decision  Spt 

•  A&S  Constructive  Environment 

•  Distributed  Mission  Operations 

•  Accelerated  Acquisition 

•  Concept  Initiatives/Demos 

•  Experiments/Tech  Transition 

Building  the  Community 

Concept  Visualization 

•  Publicizing  M&S  Story 

•  Portal  into  M&S  Exercises  & 

•  Education/Outreach 

Activities 

•  Prof  Development 

•  C4ISR  Demos  and  Trials 

•  Adv  Distr  Learning 

•  Community  Forums 

•  New  Technology  Demos 

Integrity  -  Service  -  Excellence 


3 


Air  Force  M&S  Thrusts 


U.S.AIR  FORCE 


V 

Preparing, 
Equipping  & 
Transforming 


Integrity  -  Service  -  Excellence 
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Air  Force  -  Integrated  Collaborative  Environment  (AF-ICE) 


Decision-Quality 
Analytical 
Information 
For  War-Winning 
Capabilities 


Analytical 

Needs 


Legend 

Gaps/ 

Questions 

Analytical 

Needs 


Decision 

Quality 

Information 

Analysis 


Begin  with  the  End  in  Mind . . .  T&E 


U.S.AIR  FORCE 


Pull  tools 
forward  for 
application  across 
the  Life  Cycle 


Supports  continuous 
assessment 
during  development 
and  test 


Design 
for  test 


Goal  -  Reuse  data  across  the  acquisition  timeline... 

reduces  risk  and  cycle  time 


Integrity  -  Service  -  Excellence 
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Joint  Mission  Environment 
Test  Capability  (JMETC) 


U.S.AIR  FORCE 


Integrity  -  Service  -  Excellence 
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Joint  Expeditionary  Force  Experiment 


U.S.AIR  FORCE 


(JEFX) 


■  JEFX  ’06:  Exploration  of  Network  Centric 
Operations  (NCO) 

■  9  Air  Force  Initiatives  (ConstellationNet,  AOC 
Fusion,  Homeland  Defense,  etc) 

■  5  Army  Initiatives  (Future  Combat  Systems) 

■  1  Navy  Initiative  (Global  Hawk  Maritime 
Demonstration) 


■  Theater  Battle  Management  Core  Systems  (TBMCS) 
Version  1.1.4  Development  Test 

■  Theater  Battle  Operations  Net-centric 
Environment  (TBONE) 

■  Spirals  Nov  05,  Jan  06,  Mar  06 

■  Execution  Apr-May  2006 


■  JEFX  ’08  Theme  to  Focus  on  Joint  Command  and 
Control 


Integrity  -  Service  -  Excellence 
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U.S.AIR  FORCE 


Joint  Test  Environment 
Methodology  (JTEM) 


Conduct  Joint  Mission 
Effectiveness  Analysis 


Integrity  -  Service  -  Excellence 
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Joint  Command  and  Control  of 
Net-Enabled  Weapons  (JC2NEW) 


U.S.AIR  FORCE 


■  Weapon  In-flight  Tracking 

■  Strike  De-confliction  and  Synchronization 

■  Health,  Arming  Status,  Location,  Target 
Acquisition, ... 

■  Weapon  In-flight  Target  Update 

■  Key  For  Moving  and  Mobile  Targets 

■  Key  Enabler  for  Interdiction  Role 

■  Weapon  Retargeting 

■  Importance  Increases  with  Standoff 

■  Enhanced  Time-Sensitive-Target  Capability 

■  Weapon  Impact  Assessment  (WIA) 

■  Post-strike  Assessment  and  Re-strike  Decisions 

■  Weapon  Abort 
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Net-enabled  weapons  complete  the  “Kill-Chain”  -  from  desired  actions  to  direct  effects  on 
the  battlefield,  while  providing  the  conduit  for  near-real-time  feedback  to  decision  makers 


Net-Centricity 


U.S.AIR  FORCE 


Line  of 
Sight 
STOVE-PIPES 


Connectivity 


NO  WARRIOR  FIGHTS  ALONE! 


Platform-Cen trie  Operations 

Limited  Situational  Awareness 
Limited  Real  Time  Control 
Limited  En  Route  Planning 


Net- Centric  Operations 
Distributed  Forces 

Family  of  Integrated  Operational  Pictures 
Real  Time  Situational  Awareness 
Precision  Engagement 
En  Route  Planning  &  Targeting 


The  Ultimate  End  State  is  Decision-Centric  Operations 
Employing  a  Network-Centric  Infrastructure 
“NETWORK-ENABLED  WARFARE” 


w 


Decision-Centricity 


U.S.AIR  FORCE 


Joint  Integration  through  the  Global  Information  Grid 


Arn,y  Coalition 

C2&ISR  C2&ISR 


Centers 


Centers 


Services 
&  Coalition 
DCGS 


Services 
&  Coalition 
DCGS 


.Ramstein 


Langley 


Deployed 

Sites 


Beale 


Falconer 


Hickam 


ANG 

Remotes 


Remotes 


NIMA 


ASOC 


Transformation 


SPACEAF 


WOC  EOC 


National  SIGINT 


SBIRS  PLUS 


Commercial  IMINT 


National  IMINT 


Global  Hawk 


LANDWARnet 


FORCEnet 


Predator 


UCAV 


Rivet  Joint 
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U.S.AIR  FORCE 


Testing  in  a  Joint  Environment 


IOC 


FOC 


JCIDS 


Concept 

Technology 

System  Development 

Production  &  Deployment 

Operations  & 

.  Refinement 

Development 

&  Demonstration 

A  FRP 

Support 

s/  Concept 

0 

LRIP/IOT&E  ''  Decision 

^  Decision 

Critical  Design  Review 

Review 

Evaluation 

Continuum 


jroc  Pre-  Svstems  Acquisition 

Systems  Acquisition 

Sustainment 

Live 

vfosionEnvironment  — — — 

Constructive 

— 

Concept  Dev  Sys  Design  System  Dev  DT&E  IOT&E  OT&E  Training  Ops 


Integrity  -  Service  -  Excellence 
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Raytheon 

Customer  Success  Is  Our  Mission 


T&E  M&S  War  Game  Results 


Jim  Hazlett 

Director,  Mission  Profiling 
Mission  Innovation 

Raytheon  Integrated  Defense  Systems 


Game  Overview 


Raytheon 

Integrated  Defense  Systems 


•  Purpose: 

-  Recent  seminar  war  game  for  modeling  and  simulation  (M&S)  sponsor  to 
identify  areas  for  M&S  strategic  investments  in  support  of  future  force 
developmental  and  operational  test  and  evaluation  (T&E) 

•  Participation: 

-  About  30  players  from  T&E,  M&S,  developmental  &  operational  test  & 
evaluation  (DT&E  /OT&E)  and  other  communities 

-  Divided  into  three  interdisciplinary  teams 

-  Game  research,  design,  facilitation  and  reporting  provided  to  players 

•  Game  Research: 

-  Purpose: 

Understand  the  challenges 
Gather  appropriate  issues  for  game  discussion 
Build  credible  scenarios 
Identify  subject  matter  experts 

-  Generate  interest  in  game 

-  About  15  DoD,  joint  and  service  commands  and  organizations  visited 
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Game  Scenarios  and  Moves 


Raytheon 

Integrated  Defense  Systems 


•  Scenarios  (2015  -  2020): 

-  Provide  participants  with  a  context  for  discussion  -  not  an  operational 
war  game 

-  Three  representative  scenarios  (linked  to  Defense  Planning  Guidance 
(DPG)  scenarios)*: 

Urban  canyon 
Triple  canopy  jungle 
Rugged  terrain 

•  Four  moves  (Given  a  Future  Blue  Force): 

-  Move  1 :  Red  Move  -  Counter  the  Blue  Force 

-  Move  2:  Blue  Force  Evaluation  Metrics 

-  Move  3:  Test  the  Blue  Force  -  Identify  needed  T&E  capabilities  and 
specific  M&S  challenges 

-  Move  0:  Characterize  the  barriers  between  present  and  future  state 
and  capture  T&E,  M&S  strategic  priorities 

*  All  with  Deep  Underground  (DUG)  facilities 
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Game  Approach 


Raytheon 

Integrated  Defense  Systems 
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Game  Move  Design 


Raytheon 

Integrated  Defense  Systems 


Move  1 

Move  2 

Move  3 

Move  0 

Scenario  A 

Team  A: 

Red-team  Blue 
force 

Team  A: 

Develop  Blue 
force  metrics 

Team  A: 

ID  T&E  caps 
and  M&S 
challenges 

Team  A: 

Strategic 

priorities 

Scenario  B 

Team  B: 

Red-team  Blue 
force 

Team  B: 

Develop  Blue 
force  metrics 

Team  B: 

ID  T&E  caps 
and  M&S 
challenges 

Team  B: 

Strategic 

priorities 

Scenario  C 

Team  C: 

Red-team  Blue 
force 

Team  C: 

Develop  Blue 
force  metrics 

Team  C: 

ID  T&E  caps 
and  M&S 
challenges 

Team  C: 

Strategic 

priorities 

Tactical 


Operational 


Strategic 
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Level  of  Use 


Competing  T&E  M&S  Paradigms 


Raytheon 

Integrated  Defense  Systems 


•  The  currentfold”  paradigm  of  reduced  modeling  and 
simulation  as  hardware  comes  on  line  will  no  longer  apply  as 
M&S  matures  and  as  T&E  integrates  with  other  systemic 


M&S 


The  Old  Paradigm  of  Modeling  and  Simulation  vs.  Hardware  T&E 
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The  T&E 
“Virtuous  Spiral” 


Recommended  Actions  (1) 


Raytheon 

Integrated  Defense  Systems 


•  Develop  a  mapping  of  M&S  capabilities  into  T&E  requirements 

•  Develop  plans  to  better  integrate  M&S  into  design  and  T&E  of  future 
forces 

•  Create  a  DoD  T&E  and  M&S  professional  society  to  facilitate  T&E  and 
M&S  exchange  and  interactions  and  to  foster  development  of  next 
generation  of  T&E  and  M&S  professionals 

•  Integrate  M&S  into  development  and  acquisition  programs  in  a  value- 
added  manner 

•  Organize  Electronic  Attack  (EA)  T&E/M&S  workshops  to  address  how 
EA  and  other  network  disruptions  are  addressed  in  programs,  T&E  and 
M&S 

•  Examine  how  T&E  and  M&S  is  being  employed  by  specific  programs  to 
address  C4ISR  backbone  issues 

•  Define  situational  awareness/understanding  and  sensor  fusion 
representation  requirements  for  developmental  and  acquisition  programs 
and  for  T&E  and  M&S  communities 
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Recommended  Actions  (2) 


Raytheon 

Integrated  Defense  Systems 


•  Work  with  National  Geospatial-Intelligence  Agency  (NGA)  to  develop 
common  set  of  terrain  requirements  and  representations  to  support 
programs  and  T&E 

•  Promote  improved  collaboration  between  T&E  and  M&S  community  and 
joint/service  Red,  intel,  and  doctrine  development  (including  joint). 

•  Integrate  M&S  into  development  and  acquisition  programs  in  a  value- 
added  manner  to  facilitate  integrated  development  and  T&E 

•  Invest  in  M&S  representation  of  sensor  fusion  to  support  system 
development  and  T&E 

•  Examine  use  of  M&S  in  development  of  future  force  with  an  eye  toward 
integrating  it  in  such  a  way  that  it  optimally  supports  system  development 
and  T&E  (i.e.,  Improvised  Explosive  Devices  (lEDs) 

•  Institute  DoD  M&S/T&E  professional  organization  to  support  information 
exchange,  coordination,  and  community  replenishment  and  growth 

•  Streamline  and  update  DoD  testing  security  protocol  to  keep  current  with 
changing  requirements  to  protect  developmental  capabilities 

•  Work  with  T&E  and  community  to  collaborate  on  various  visions, 
paradigms  and  versions  of  future  programs 
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Major  Strategic  Findings 


Raytheon 

Integrated  Defense  Systems 


•  Make  investments: 

-  M&S  of  system  of  systems  and  representation  of  C4ISR  fusion 

-  Developing  joint  M&S  collaborative  environment,  infrastructure  and 
common  framework 

-  Scalable/high-fidelity,  composable,  transformable  architecture; 

-  Robust  synthetic  natural  environment 

i.e.,  wave  propagation  and  terrain  modeling 

•  Develop  capabilities 

-  Provide  replication  of: 

Equipment  and  systems 
Trained  personnel 
Territory 

-  Capture  secondary,  systemic  effects  of  equipment  and  system  changes 

-  Experiment  with  new  doctrine  and  developmental  capabilities  -  e.g., 
asymmetric  warfare 
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Test  and  Training  Enabling  Architecture  (TENA) 
Offers  Range  Interoperability  and  Resource  Reuse 
Solutions  for  Test  and  Training  Ranges 


Gene  Hudgins 

TENA  User  Support  Lead 


NDIA  Test  &  Evaluation  Conference,  Jacksonville,  FL 


rENA  Mission 


Currently,  range  systems  tend  to  be  non-interoperable, 
“stove-pipe”  systems 

The  purpose  of  TENA  is  to  provide  the  architecture  and  the 
software  implementation  necessary  to 

•  Enable  Interoperability  among  Range  systems,  Facilities, 
Simulations,  C4ISR  systems  in  a  quick,  cost-efficient  manner, 
and 

•  Foster  Reuse  for  Range  asset  utilization  and  for  future 
developments 


•  Support  the  Warfighter  (Joint  Vision  2010/2020) 

•  Enable  Simulation-Based  Acquisition 

•  Foster  Test  and  Training  Integration 

•  In  the  long  term:  SAVE  MONEY! 


Lay  the  Foundation  for  Future  Test  and  Training  Range 

Instrumentation 


CTEIP  Integrated  Architecture 
Global  Command  and  Control 

Network 


Control  Center 


Global  Positioning 
System 


*  Realistic  Testing 

*  Joint  Testing 

*  Continuous  Testing 

*  Enhanced  Test 
Capability 


Satcom 


Satcom 


High  Powered  Computer 
Modeled  Radar 
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TENA  is  a  Test  and  Training 
Interoperability  Enabling  Technolog 


Remote  Ground 

Station 


■  PC 


PS  INTERNAL 
SUBSYSTEM  (IS) 


TENA  Software  Development 
Activity  (TENA  SDA) 


•  TENA  Software  Development  Activity  (TENA  SDA)  will  assume 
sustainment  and  future  development  responsibilities  for  TENA  for 
both  the  test  and  training  communities 

•  Reports  to  CTEIP  and  the  JNTC  Joint  Management  Office  (JNTC 
JMO)  on  all  TENA-related  activities,  including  but  not  limited  to: 

•  Sustainment  of  TENA  Middleware 

•  Ports  to  different  operating  systems 

•  Upgrades  to  the  TENA  Middleware 

•  Upgrades  to  TENA-related  tools  and  utilities  (such  as  the  auto-code  generator) 

•  Distribution  of  TENA  Middleware 

•  Distribution  of  source  code  generated  from  object  models 

•  Correction  of  software  defects 

•  Technical  support  to  TENA  users,  including  on-line  help  desk  and  TENA  Training 

•  Upgrades  to  TENA  capabilities  will  stem  from: 

•  Inputs  from  the  Services  (including  from  the  annual  reports  the  Services  provide 
on  their  implementation  of  TENA  on  their  systems) 

•  Inputs  from  the  T&E  Executive  Agent  Needs  and  Solutions  process 

•  Joint  training  requirements  through  the  JNTC  JMO 

•  Common  requirements  identified  by  members  of  the  TENA  AMT 

•  Feedback  provided  by  TENA  users 

•  Results/observations  from  test  and  training  events 

•  Other  responsibilities  include  chairing  the  TENA  AMT 


Where  TENA  SDA  Fits  in  DoD 


Gen  Counsel 

ATSD  Intel  OVst 

DoD  IG 

ATSD  Civ  Spt 

Secretary  Of  Defense 
Deputy  Secretary  Of  Defense 
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i 


i 


USD  Policy 

USD  Comp 

USD  P&R 
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USD 
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Office  Of  The  Secretary  Of  Defense  (OSD) 


Sec  Navy 
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dant 

MC 

Defense  Agencies  -I—  DOD  Fid  Activities 
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Congress 
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AF 
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Ch  ofStf 
Army 

DCAA 

DCMA 

DFAS 

DISA 

DIA 

DLA 
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DTRA 
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AFIS 

DefPOW/MP  Office 
DoD  Edu  Activity 
DoD  HR  Activity 
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Adjustment 
TRICARE  Mgt 
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Wash  Hq  Service 


Chairman  JCS 


Vice  Chairman 


Joint  Staff 


CoS  Army 
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CoS  AF 

Commandant  MC 


|  DP  SE  | 

| DDT&E | 
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TENA  Software  Development 
Activity  (SDA)  Objectives 

•  Refine  and  sustain  the  common  Architecture  for  the 
test/training  range  community  -  “TENA”  (Test  &  Training 
Enabling  Architecture) 

•  Define  a  common  Object  Model  to  be  used  across  the  ranges 

•  Continue  development  of  a  common  Software  Middleware  that: 

•  Uses  the  object  model 

•  Enhances  interoperability  and  reuse  among  the  ranges 

•  Refine  common  understanding  of  range  processes 

-  the  Logical  Range  Concept  of  Operations 

•  Define  and  prototype  common  Tools  to  configure  and 
conduct  multi-range,  synthetic  test  events  or  training 
exercises 

•  Create  distributed,  synthetic  battlespaces  with  real  weapon  systems 

•  Link  multiple  ranges  together  to  form  a  larger,  cohesive  range 

•  Enable  testing,  assessment,  experimentation,  and  training  of  weapon 
system  interoperability,  C4ISR,  and  system-of-systems 


Architecture  Management  Team 

(TENA  AMT) 


System  Engineers  &  Technical  Leads  for  the  current  major  stakeholders  of 
TENA 


N. 


Meetings  every 
6-8  weeks 


AAC,  Eglin  AFB  FL 
NUWC,  Newport  Rl 
RTTC,  Huntsville  AL 
PMRF  Synthetic  Range 
EPG,  Fort  Huachuca  AZ 
WSMR,  White  Sands  NM 
NAWC-AD,  Pax  River  MD 
P5  Combat  Training  System 
Virtual  Proving  Ground  (VPG) 

Joint  National  Training  Capability  (JNTC) 

NAWC-WD,  China  Lake  &  Point  Mugu  CA 
Next  Generation  Range  Instrumentation  (NexRI) 

New  Generation  Targetry  System  (NGATS) 

Enhanced  Range  Application  Program  (EnRAP) 

NAVSEA  Warfare  Center  -  Keyport,  Keyport,  WA 
Common  Training  Instrumentation  Architecture  (CTIA) 

Army  Operational  Test  Command  (OTC),  Fort  Hood,  TX 
NAVAIR  Tactical  Training  Ranges  Program  Office  (PMA-205) 


„  Boein9’ 
MIT  LL> 

sAlC’  nMSO,  NRL’  , 
JlTC-  also 

VItenSdC&&PartiC'pate> 


■  Design  Decisions  /  Trade-offs  /  Status 

■  TENA  Use  Cases  /  Prototype  Test  Strategies 

■  Technical  Exchanges  of  Lessons  Learned 

■  Issues  &  Concerns  Identification,  Investigation,  &  Resolution 
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TENA  Was  Developed  in  Spirals 
with  the  Ranges  Involved 


Test  & 
Training 
Enabling 
Architecture 
(TENA) 

TENA  was  revised  based  on  user  feedback  and  lessons  learned  from 
working  software  prototypes 

TENA  will  continue  to  evolve  based  upon  emerging  requirements 
TENA  users  (via  AMT)  determine  what  functionality  is  added  to  TENA 

TENA  is  based  on  real-world  tests  at  real  ranges 
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TENA  Architecture  Overview 


TENA  Applications 
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TENA  Tools 
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Non-TENA  Applications  -sMe  10 


TEN  A  Architecture  Overview 
with  Current  Efforts  Highlighted 


•  MagicDraw  Plugin  for  TDL 

TEN) 

|~i  TENA  IDE 

•  JRF05  LROM  Tool 

Non-TENA 
System 

>plications 
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Ways  TENA  Middleware  Can 
Exchange  Data 


■  TENA  presents  to  the  range  user  a  unification  of  several 
powerful  inter-application  communication  paradigms 

■  Publish/Subscribe 

Similar  in  effect  to  HLA,  DIS,  or  other  PDU-based  communication  systems 

Each  application  publishes  certain  types  of  information  (the  publication  state) 
which  can  be  subscribed  to  by  any  other  application 

■  Remote  Method  Invocation 

Similar  to  CORBA  or  Java  RMI 

Each  object  that  is  published  may  have  methods  that  can  be  remotely  invoked 
by  other  applications 

■  Messages 

Individual  messages  that  can  be  sent  from  one  application  to  one  or  more 
other  applications 

■  Data  Streams 

Native  support  for  audio,  video,  telemetry,  and  tactical  data  links 
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Data  Streams  Demonstrated 
at  Recent  AMT  Meeting 


■  TENA  provides  remote  control  of  data  streams 

■  Allows  COTS/GOTS  (such  as,  third-party  vendor)  streaming 
solutions  and  technologies  to  be  used 

■  TENA  approach  promotes  interoperability  and  reuse  by 
standardizing  software  interfaces  and  supporting  the 
packaging  of  server/client  stream  components 


Live  Video  Stream  Transmitted 
over  Wireless  Network 


Video  Stream  File  Played  Back 
over  Wireless  Network 
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Capabilities  of  DIS,  HLA, 

and  TENA 


■  DIS  only  provides  network,  “on-the-wire”  standard 

■  HLA  provides  some  services  and  capabilities 


Notification 

Services 


Real  Software 
Objects 


itandard  TEN; 
Object  Model 


Repository 


Gateways  to 
Other  Archs. 


Data  Streams 


Usable 

Interface 


Data  Collectors 


Persistent 
Data  Mgmt 


Local  Methods 


Open-Source 

Software 


Interfaces 


Standard 

Tools 


Federation 

Management 


Operational 

Architecture 


Compile-Time 
Type  Safety 


Object  Model 
Utilities 
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- - - (  Distribution 

Viyianagemei^ 


vlulti-Threadec 

Infrastructure 


Compiled-in 
Object  Model 


Marshaling/ 
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Time 

Management* 


TENA 


TENA  provides  more 


*Time  Management  can  only  be  used  in  simulation-only  events. 
If  any  live  systems  are  involved  (as  is  the  case  in  all  HWIL  and 
range  events),  time  management  can  not  be  used 


Joint  Forces  Command  (JFCOM) 

Use  of  TENA 


■  Live  Data  Instrumentation  Infrastructure 

■  TENA  serves  at  JNTC  integration  architecture  for  range 
interoperability  and  bridge  to  simulation  network 

■  Progressive  support  to  JFCOM/JNTC  Events: 

■  Millennium  Challenge  2002  (MC-02) 

TENA  provides  common  data  model  via  gateways  to  integrate  Range 
Instrumentation  into  JTASC  GCCS 

.  JCIDEX-03 

Enhanced  data  model  and  native  TENA  interfaces  for  Range 
Instrumentation  and  Analysis  Systems  for  JCID  and  RTCA  assessment 

.  WRC  Horizontal  Thrust  Event  (HTE) 

TENA  Application  Management  Object  implemented  to  control  Range 
Instrumentation  data  feeds  and  integrate  for  JCAS  assessment 

.  CJTFEX-04 

Reuse  of  data  model  and  native  TENA  interfaces  for  Range  Instrumentation 
and  Analysis  Systems  for  JCID  and  JT&E 

■  Joint  Red  Flag  2005  (JRF-05) 

Combines:  Red  Flag  05,  Virtual  Flag,  Roving  Sands  05,  Battle  Group  Inport 
Exercise  (BGIE),  Joint  Systems  Training  Exercise  (JSTE) 
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Range  Integration  in  Millennium 
Challenge  2002  (MC02) 


Blue  Forces 


Nellis  AFB 


Land  Range/China  Lake 


TEN  A  Gateway 


Ships 

Ground  forces 
•  Aircraft 

Opposing  Forces 


National  Training 
Center/Ft.  Irwin 


Electronic  Combat 
Range/China  Lake 


Aircraft  &  air  targets 
Ships  I 

Ground  forces  ^ 


Sea  Range/Point  Mugu 


TENA  Gateway 

i 

US  Marines/So.  California 
Logistics  Airfield 


Joint 
Network' 


Model  &  Simulation 
Feed 


Joint  Training , 
Analysis ,  and 
Simulation  Center 


TENA  Gateway 


Integrating 

Software 


Command,  Control, 
Communications,  Computers, 
Intelligence  Feed 


Global 
Command  & 
Control  System 
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TENA  Use  in  JCIDEX  03 


JNTC  Horizontal  Thrust  Event 
Range  Integration  Solution 


WRC 
Horizontal 
Event  DISA 
DAT  MS 
Network 


TEN  A  Server 


JTASC  WRC  Event  Network 


JTASC 


Twentynine  Palms 
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Joint  Red  Flag  2005 


CDL:  Common  Data  Link 
PCDS:  Personal  Computer  Debriefing  System 
UAV:  Unmanned  Aerial  Vehicle 
STG:Static  Target  Generator 

ADOCS:  Automated  Deep  Operation  Coordination  System 

MTI/FTI:  Moving  Target  Indicator/Fixed  Target  Indicator 

RTCA:  Real  Time  Casualty  Assessment 

JWinWAM:  Joint  Windows  based  Warfare  Assessment  Model. 

JCI:  Joint  CDL  Interface  (TENA  ARDS/ARDS-Lite  interface) 

TENA-E:  Engineering  Analysis  and  Diagnostics 

GALE:  Graphical  Area  Limitation  Environment 

MAnTSS:  Modular  Analysis  and  Test  Support  Software  (White 

Cell) 

TACO:  TACtical  Office 

CGS:  Ceirrmon-Gfound  Station  — — - - - — - - - 


Systems  Using  TENA  in  JNTC 

Exercises 

•  Advanced  Range  Data  System  (ARDS  and  ARDS-Lite)  instrumentation  interface 

•  Air  Force  Air  Warrior  instrumentation  interface 

•  Army  National  Training  Center  Instrumentation  System  (NTC-IS)  instrumentation  interface 

•  Army  Test  and  Evaluation  Command  engagement  adjudication  workstation  (named  the  Common  Data  Link  -  CDL) 

•  Automated  Deep  Operation  Coordination  System  (ADOCS) 

•  GALAXY  A  LATR-type  system  used  by  the  United  Kingdom 

•  JNTC  Live-to-Simulation  gateway  TENA  to  High  Level  Architecture  application  (named  GOTH) 

•  Joint  Close  Air  Support  Joint  Test  &  Evaluation  COMBAT  analysis  system  and  display 

•  Joint  Tactical  Information  Distribution  System  (named  RAT  TRAP) 

•  Large  Area  Tracking  Range  (LATR)  system 

•  Marine  Corps  Integrated  Global  Positioning  System  Radio  System  (IGRS)  instrumentation  interface 

•  NAVAIR  After  Action  Review  AAR/Monitor  Display  (Personal  Computer  Debriefing  System  -  PCDS) 

•  Navy  Naval  Air  Systems  Command  (NAVAIR)  RangeView  analysis  system  and  display 

•  Patriot  Program  Office,  Tactical  Office  (TACO)  analysis  and  display  system 

•  Static  Target  Generator  (STG) 

•  SureTrak  Airspace  radar  monitoring  system 

•  TENA  to  Distributed  Interactive  Simulation  (DIS)  analysis  system  (named  TOSTADA) 

•  Time,  Space,  Position  Information  (TSPI)  Internal  Entity  Re-formatter  (TIER) 

•  Warfare  Assessment  Model  (WAM)  system 
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VAST/IMPASS 
Over-the-Water  Scoring 


VAST:  Navy  Virtual  At  Sea  Training  System 
IMPASS:  Integrated  Maritime  Portable  Acoustic 
Scoring  and  Simulator  Buoy  System 


Eglin  CCF 
Eglin  AFB,  FL 
NVP,  RSCP 


CDSA 

Dam  Neck,  VA 
NVP,  RSCP 


TENA  on  Fiber 


Eglin  Range  Site  A-15 
NVP,  RSCP,  IMPASS 


GPS 

Acoustic  Processing 
Communication  Link 


TENA  on  Microwave 


NVP:  Navy  Visualization  Program 
RSCP:  Range  Safety  Control  Program 


NCSS 

Panama  City,  FL 
NVP 


Shipboard  Processing 
Map  Rendering 
Virtual  Target 


"l 


.  \  <  L 
1 1 


■  —  V 


\N 


Redstone  Technical  Test  Center 

Use  of  TENA 


TO- 


RIAB  RIAB  RIAB 

Gateway  Monitor  Control 

Computer 


X 


Tactica 


i  a 


R  AB 


issue 


MARDEC  Support  Active  Protection  System  (APS) 
■  FCS  APS  Candidate 

“Serial”  Connection  to  RIAB 

TENA  Control&  Monitor 

Configuration  Control  in  Range  Software 

Data  Logging  via  ILH  Object 
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SIMDIS  UseofTENA 


SIMDIS 


SCORE  TSPI  Feed 

Southern 
California 


NRL 

Washington,  DC 


TENA 


■  Duration  testing  using  SCORE  TSPI 

■  Four  consecutive  days 

Win  XP,  Red  Hat  9,  Solaris  5.8 
Processed  180,000+  entities 


data 


feed 


■  Two  consecutive  days 
Win  XP,  Red  Hat  9 
Processed  53,000+  entities 


■  Results  and  observations 

■  No  issues  with  discovery  latency 

■  No  issues  with  update  latency 

■  No  issues  with  CPU  usage 

■  No  issues  with  memory  usage 


Seeker  lo  Warship 
Gain:  e.ZBtdB},  pwr:  •mK(dBm) 


SIMDIS 


Threat  Systems  Test 
of  TENA 


G75  “Giraffe”  Radar  Simulation  Target  Simulation  G75  “Giraffe”  Radar  Simulation 


■  Testing  and  analysis  by  Scientific  Research  Corporation  (SRC) 

■  Results  and  observations: 

■  TENA  middleware  appears  stable  and  predictable 

■  TENA  object  model  format  is  sufficient  for  representation  of  threat  systems 

■  TENA  provides  satisfactory  functionality  and  performance  to  be  utilized 
within  a  threat  simulation  scenario  and  for  fielding  threat  simulations  Slide24 


NetAcquire  Using  TENA 

Real  Time  Embedded  Instrumentation 


■  Direct  hardware  interfaces  not  standard  on  COTS  desktops 

■  Aerospace  serial  I/O  formats  (synchronous,  telemetry,  special 
protocols,  etc.) 

■  GPS  (time  and  position) 

■  Analog  input  /  output 

■  Digital  and  pulse  input  /  output 

■  IRIG  timing 

■  Avionics  buses  (1553,  ARINC,  1394) 

■  GPIB  (IEEE-488)  instrumentation 

■  Inexpensive,  ruggedized,  mobile  form-factor 

■  Accomplishments: 

■  Took  NetAcquire  only  1 1  days  to  port  TENA  into  their  products 

■  Direct  synchronous  serial  hardware  interface  to  FPS-16  radar  system 

■  Little  or  no  programming  required  to  support  other  radar  data  formats 

■  NetAcquire  runs  a  true  real-time  operating  system,  device 
drivers,  and  application  software 

■  Provides  TENA  with  deterministic  and  bounded  response  times 
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The  Way  Ahead  for  TENA 


•  Continue  partnership  Joint  National  Training  Capability  (JNTC) 

•  Use  the  JNTC  and  JNTC-like  events  to  reduce  risk  and  refine  application  of  TENA 

•  Weapons  Tactics  &  Instruction  (WTI  06-2) 

•  Terminal  Fury 

•  Talisman  Saber 

•  Joint  Mission  Environment  Test  Capability  (JMETC) 

•  Army  Cross  Command  Collaboration  (3CE) 

•  InterTEC 

•  10  Range 

•  Cope  Thunder 

•  Technically  support  and  partner  with  PMs  in  their  assessment  and 
implementation  of  TENA  for  Test  and  Training  applications 

•  Use  the  current  TENA  Requirements-Driven  and  Stakeholder- 
Prioritized  process  to  spiral  develop  and  prototype  further  TENA 
capabilities 
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TENA  Training  Available 


■  TENA  Technical  Overview  Course  (TOC) 

■  Designed  for  the  non-programmer 

■  Provides  basic  familiarization  on  TENA  and  Logical  Ranges 

■  Lecture  format  (full  day,  half  day,  and  two-hour  versions  available) 

■  TENA  Technical  Introduction  Course  (TIC) 

■  One  day,  lecture  class  for  software  programmers 

■  Introduces  design  concepts  to  build  TENA-compliant  applications 

■  14  classes  held  to  date 

■  More  than  350  software  programmers  trained  to  date 

■  Classes  held  at  White  Sands,  Point  Mugu,  RTTC,  Eglin,  Orlando, 
Alexandria,  and  London 

■  TENA  Middleware  Hands-on  Training  (HOT) 

■  Four-day,  computer  class  for  software  programmers 

■  Provides  several  examples  &  exercises  to  learn  the  TENA  Middleware  API 

■  12  classes  held  to  date 

■  More  than  250  software  programmers  trained  to  date 

■  Classes  held  at  White  Sands,  Point  Mugu,  RTTC,  Eglin,  Alexandria, 

China  Lake,  and  Dugway  (Salt  Lake  City)  slide 


Obtaining  &  Installing  the  TENA 
Middleware  and  Object  Models 


•  Get  Middleware  from  the  website 


•  http://www.tena-sda.org,  log  in,  go  to  TENA  Middleware  Support 


Registered  User  Account  required 


TENA  -  Test  and  Training  Enabling  Architecture 


; 


TENA  Website 
aHome 

BTENA  Information 
BTENA  Middleware  Support 
aTENA  Object  Model  Support 
B  Website  Support 
B  TENA  Team  Resources 
B  Deployment  Support 


This  web  site  is  the  Test  and  Training  Enabling  Architecture 
(TENA)  portal. 

Developed  under  a  joint  interoperability  initiative  within  the  Department  of  Defense,  TENA  is  enabling 
interoperability  among  ranges,  facilities,  and  simulations  in  a  quick  and  cost-efficient  manner,  and  fostering 
reuse  of  range  resources  and  range  system  developments.  TENA  is  managed  by  the  TENA  Software 
Development  Activity  (TENA-SDA). 


For  a  TENA  audio  & 

niHnn  nrnnnMhnn' 

This  website  is  the  Test  and  Training  Enabling  Architecture  (TENA)  portal. 


Note:  This  site  makes  extensive  use  of  "pop-ups"  and,  thus,  requires  this  feature  to  be  enabled  in 
your  browser  for  complete  functionality! 


The  menu  below  has  information  available 
to  anyone  interested  in  TENA.  Once  an 
account  has  been  requested  (using  the 
registration  link  at  the  bottom  of  the  page) 
and  approved;  other  options  wiW  become 
available.  Use  the  login  on  the  right  to  go 
to  the  controlled  portion  of  the  website. 

TENA  Website  Menu 

0  TENA  Information 

0  TENA  Middleware  Support  (download  acct  req'd) 
0  TENA  OM  Support  (download  acct  req’d) 

0  Website  Support  (web  acct  req  'd) 

For  a  TENA  audio  &  video  presentation: 


»  Request  Account 
»  Password  Reminder 
»  Username  Reminder 
»  Contact  Info 


Developed  under  a  joint  interoperability  initiative  within  the  Department  of  Defense, 
TENA  is  enabling  interoperability  among  ranges,  facilities,  and  simulations  in  a  quick 
and  cost-efficient  manner,  and  fostering  reuse  of  range  resources  and  range 
system  developments.  TENA  is  managed  by  the  TENA  Software  Development 

Activity  (TENA-SDA). 


Username: 


Password: 


- TEST  AND  TRAINING 

ENABLING 

ARCHITECTURE 


TENA  ARCHITECTURE 

•  TENA  Architecture  Reference  Document 


LATEST  NEWS 

•  TENA  Release  5.1  available  for  download  (1  June  2005) 

•  TENA  Used  Extensively  in  Joint  Red  Flag  2005  (9  May  2005) 

•  New  MagicDraw  TDL  Generator  plugin  available  for  download  (4  April  2005) 

•  Platform  OM  Technical  Exchange  January  2005  Material  (27  Jan  05) 


NEW  EVENTS  &  REGISTRATION 

•  Register  for  TENA  Technical  Introduction  Class,  August  5,  2005,  8  A.M.  -  3  P.M.,  in  Portsmouth,  Virginia 

•  Register  for  AMT-29  on  3-4  August  2005  in  Portsmouth,  VA. 

•  TENA  Tutorial  to  be  presented  11  July  at  Annual  ITEA  Technology  Review,  Atlanta,  GA.  Register  at 
http://www.itea.org 


WEBSITE  NEWS 

•  You  can  now  request  TENA  middleware  download  privileges.  NOTE:  Users  accounts  are  provided  to  individuals 
and  account  sharing  (i.e.,  providing  the  account  password  to  others)  is  strictly  forbidden. 


Today's  date  is  2005-06-30 
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Downloading  the 
"ENA  Middleware 


TEN  A  -  Test  and  Training  Enabling  Architecture 


epowell- 
approved: 
profile  |  logout. 

it 


Browse  Repository  I  Add  Object  Model 


Middleware: 


Platform 


TENA-vS.I 
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File  Download  -  Security  Warning 


Filename 


fc3-£cc343-d 


rENA-fc3-gcc343-d-v5-l.bin 


fc3-2cc343 


TENA-fc3-acc343-v5-l.bin 


irix65-niipspro742m-d  1  ENA-irix65  -mipspro  742m-d-v5 . 1  .bin 


irix65-mipspro742m  TENA-irix65-mipspro742m-v5 . 1  .bin 


rh8-scc32-d 


TENA-rh8-scc32-d-v5 . 1  .bin 


rh8-scc32 


TENA-rh8-scc3  2-v5 . 1  .bin 


rh9-acc322-d 


TENA-rh9-acc322-d-v5 . 1  .bin 


rh9-ecc322 


TENA-rh9-scc322-v5 . 1  .bin 


Download 


Download 


Download 


Download 


Download 


Download 


Download 


nnnviimETki  - -  - ^  **  w»  _ 

Dowmoaditats  for  04/2005. 

#  ~500  middleware  down  o 


(54.496MB) 


Do  you  went  to  run  or  save  this  file? 


(27.424MB) 


(37.801MB) 


(21.75MB) 


Name: 

TENA-xp-vc7  l-d-v5[  1].  l.exe 

Type: 

Application,  59.5  MB 

From: 

www.tena-sda.org 

(132.572MB) 


(40.312MB) 
(132.661MB) 
(40,5481 


Run 


While  files  from  the  Internets 


Save 


Cancel 


rhelws4-gcc343-d  TENA-rhelws  4  -  gc  c 3  4 3  -  d-v  5 .  l.bin 


rhelws4-scc343 


TENA-rhelws4-scc343-v5 . 1  .bin 


solaris8-scc323-d 


TENA-solaris8-scc323-d-v5-l.bin 


solaris8-scc323 


TENA-solaris8-scc323-v5 . 1  bin 


2k-vc71-d 


TENA-2k-vc7 1  -d-v? .  1  .exe 


581MB) 


2k-vc71 


TENA-2k-vc7 1  -v5 . 1  .exe 


124.365MB) 


xp-vc71-d 
xp-vc7 1 


TENA-xp-vc7 1  -d-v5 . 1  .exe 


Download 


TENA-xp-vc7 1  -v5 . 1  .exe 


4512MB) 

1,361MB) 
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rENA  Supported  Platforms 


Operating  System 

Compiler  Type 

Status 

Embedded  Planet  (Linux  Real-Time  OS) 

GCC  3.2.2 

Supported  (5.1.1  release) 

Linux  -  Fedora  Core  3 

GCC  3.4.3 

Supported 

Linux  -  Fedora  Core  4 

GCC  4.0.1 

Supported  (5.1.1  release) 

Linux  -  Red  Hat  8.0 

GCC  3.2 

Supported 

Linux  -  Red  Hat  9.0 

GCC  3.2.2 

Supported 

Linux  -  Red  Hat  Enterprise  WS  4 

GCC  3.4.3 

Supported 

Phar  Lap  ETS  -  NetAcquire  (HW 
integrated  Windows  Real-Time  OS) 

MS  Visual  C++ 7.1 

Supported  (through  CRADA 
with  NetAcquire) 

SGI  IRIX  6.5.22m 

GCC  3.3 

Supported 

Solaris  8 

GCC  3.2.3 

Supported 

Solaris  10 

Sun  SPRO  5.8  (w/wo  64  bit  mode) 

Supported  (5.1.1  release) 

Windows  2000 

MS  Visual  C++ 7.1 

Supported 

Windows  XP 

MS  Visual  C++ 7.1 

Supported 

MAC  OS  10.4.2 

GCC  3.3 

Unable  to  support  with  5.1.1 
release  due  to  TAO  1 .3 
incompatibility,  will  add  when 
TAO  is  upgraded 

VxWorks  6.1 

GCC  3.3.2 

Port  in  progress 

Installing  the  Release 


TENA  -  Test  and  Training  Enabling  Architecture 


The  TENA  Middleware  Release  Notes 

Release  5.1 


Prepared  for: 


Central  Test  and  Evaluation  Investment  Program  (CTE1P) 

The  TENA  Middleware  was  developed  under  contract  1435-04-01 -CTO  1085 
in  which  the  regulations  from  DFARS  252.227-7013  and  252.227-7014  are  enforced. 

Science  Applications  International  Corporation 
Kings  towne ,  VA 


The  TENA  Middleware  Installation  Guide 

Release  5.1 


Prepared  for: 


Central  Test  and  Evaluation  Investment  Program  (CTEIP) 

The  TENA  Middleware  was  developed  under  contract  1435-04-0 1 -CTO  1085 
in  which  the  regulations  from  DFARS  252.227-7013  and  252.227-7014  are  enforced. 

Science  Applications  International  Corporation 
Kingstowne ,  VA 
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TENA  Standard  Object  Models 


•  TENA-Platform: 

•  TENA-Platform-v3.1 

•  TENA-PlatformDetails-v3 

•  TENA-Affiliation-vl 

•  TENA-UniquelD-v2 

•  TENA-PlatformType-vl 

•  DIS-EntityType-vl 

•  TENA-Munition-v2.1 

•  TENA-Engagement-v3.1 

•  TENA-Organization-vl 

•  TENA-EmbeddedSystem-v2 

•  TENA-EmbeddedSensor-v2 

•  TENA-EmbeddedWeapon-v2 

•  TENA-AMO: 

•  TENA-AMO-vl 


•  TENA-TSPI: 

•  TENA-TSPI-V4 

•  TENA-Time-vl.1 

•  TENA-Position-vl 

•  TENA-Velocity-vl 

•  TENA-Acceleration-vl 

•  TENA-Orientation-vl 

•  TENA-AngularVelocity-vl 

•  TENA-AngularAcceleration-vl 

•  TENA-ORM-vl 

•  TENA-SRF-vl 

•  TENA-SRFserver-vl 
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rENA-TSPI-v4 

(TENA  SDA  Supported) 


«TENA::LocalClass» 

TSPI 


«TENA::LocalClass» 

«TENA::LocalClass» 

«TENA::LocalClass» 

«TENA::LocalClass» 

Time 

Velocity 

Acceleration 

Orientation 

«TENA::LocalClass» 

Position 

-pi  :  TENA:: double 

-p2:TENA::double 

-p3 :  TENA::double 

-srf :  SRFenum 

-srfData  :  TENA:: double  [0.*] 

-orm :  ORMenum 

+get_GeocentricPosition(  srf:  GeocentricSpatialReferenceFrame  ) :  GeocentricPosition 

+set_GeocentricPosition(  pos  :  GeocentricPosition,  srf:  GeocentricSpatialReferenceFrame  ) :  TENA::void 

+get_GeodeticPosition(  srf:  GeodeticSpatialReferenceFrame  ) :  GeodeticPosition 

+set_GeodeticPosition(  pos  :  GeodeticPosition,  srf:  GeodeticSpatialReferenceFrame  ) :  TENA::void 

+get_LocalTangentPlaneENUposition(  srf:  LocalTangentPlaneENUspatialReferenceFrame  ) :  LocalTangentPlaneENUposition 

+set_LocalTangentPlaneENUposition(  pos  :  LocalTangentPlaneENUposition,  srf:  LocalTangentPlaneENUspatialReferenceFrame  ) :  TENA::void 

+get_LocalSphericalTangentPlanePosition(  srf:  LocalSphericalTangentPlaneSpatialReferenceFrame  ) :  LocalSphericalTangentPlanePosition 

+set_LocalSphericalTangentPlanePosition(  pos  :  LocalSphericalTangentPlanePosition,  srf:  LocalSphericalTangentPlaneSpatialReferenceFrame  ) :  TENA::void 

+get_SRF() :  SpatialReferenceFrame 


«TENA::LocalClass» 

GeocentricPosition 

«TENA::LocalClass» 

GeodeticPosition 

«TENA::LocalClass» 

LocalTangentPlaneENUposition 

«TENA::LocalClass» 

LocalSphericalTangentPlanePosition 

+x :  TENA::double 
+y :  TENA::double 
+z :  TENA::double 

+latitude :  TENA::double 
+longitude :  TENA::double 
+heightAboveEllipsoid :  TENA::double 

+x :  TENA::double 
+y :  TENA::double 
+z :  TENA::double 

+elevation :  TENA::double 
+ azimuth  :  TENA:: double 
+range :  TENA::double 
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TENA  Solutions  to 
Interoperability  Challenges 


•  On-the-Wire  Specification  vs.  API  Standard 

API  Standard  allows  future  technological  advances  for  data  transmission 
to  be  much  more  cost-effectively  incorporated 

•  Single  Reference  Frame  vs.  Multiple  Reference  Frames 

Multiple  Reference  Frames  allow  different  range  systems  to  operate  in  the 
coordinate  system  most  optimum  for  their  range 

•  Single  Level  vs.  Multiple  Levels  of  Compliancy 

Multiple  Levels  of  Compliancy  allow  a  more  meaningful  definition  of 
compliancy  to  be  used  among  Range  engineers  &  investment  managers 

•  Run-Time  Interpreter  vs.  Compile-Time  Integration 

Compile-Time  Integration  allows  for  inconsistencies  to  be  discovered 
when  the  software  is  being  upgraded  vice  during  the  event 

•  Hand-Coded  vs.  Auto-Code-Generated  Interfaces 

Auto-Code-Generated  Interfaces  can  be  produced  more  reliably  and 
tremendously  faster  than  traditional  hand-coded  interfaces 
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Summary 


\ 

TENA  is  an  Architecture  for  Ranges,  Facilities, 
and  Simulations  to  Interoperate,  to  be  Reused,  to 
be  Composed  into  greater  capabilities 

* 


■  TENA  can  be  downloaded  from  the  Web  (for  free) 

■  TENA  Middleware  currently  works  on  Windows,  Linux,  and  Sun 


■  Users  are  involved  in  the  process  to  develop  and 
expand  the  architecture 

■  CTTRA  Workshops,  AMT  Meetings,  and  RCC  Coordination 


■  TENA  is  the  JNTC  architecture  for  Live  integration 

■  TENA  is  being  used  in  a  number  of  applications 
including  vendor  instrumentation  systems 
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Important  Contact  Information 


•  Project  Website:  http://www.tena-sda.org 

•  Download  TENA  Middleware 

•  Submit  Helpdesk  Case  (http://support.fi2010.org) 


•  TENA  Architecture  Reference  Document 

•  http://www.tena-sda.org/documents/tena2002.pdf 


•  TENA  Feedback:  feedback@tena-sda.org 

•  Provide  technical  feedback  on  TENA  Architecture  or  Middleware 

•  Ask  technical  questions  regarding  TENA 

•  Provide  responses  to  AMT  action  items 

•  Request  TENA  training 
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Non-TecnnlcaJ  Obstacles  to 


r*r* 


crrectlve  lvJ03  Collectivity  jri 


^tanel  J|hairf^Mrf  John  Igigen 

Gruffraa n  S'^ulation  Technologies 


Introduction 


■  Greetings 

■  Introduction  of  Panel  Members 

■  John  lllgen,  Northrop  Grumman 

■  Dr.  Paul  Deitz,  ARL 

■  Mr.  Rick  Cozby,  DTC 

■  Mr.  Jack  Sheehan,  FCSCTO 

■  Mr.  Augie  Ponturiero,  Northrop  Grumman 


A  Changing  Landscape 


■  Capabilities-Based  Acquisition 

■  Warfighter-focused 

■  New  thought  processes  on  WHY,  WHAT,  and  HOW 
to  develop  new  Systems/Families  of 
Systems/Systems  of  Systems 

■  New  Systems/Families  of  Systems  “Born  Joint” 

■  Services  working  together  at  all  levels  to  bring 
capabilities  to  the  Warfighters 

■  Funding  Constraints 

■  New  Developments  vs  Current  Operations  Support 

■  Using  more  Virtual  and  Constructive  entities  for 
testing 

M&S  more  significant  in  Test  and  Evaluation 


The  Effect 


■  Increased  cooperation  between  Services  and  DoD 
Agencies  at  all  levels 

■  Multi-Service  and  Joint  Testing 

■  OSD-driven  -  “Testing  in  a  Joint  Environment 
Roadmap” 

■  What  is  the  impact  to  Service  Acquisition  efforts? 

■  PM,  T&E,  Warfighters  must  work  together  across 
Service  boundaries  to  field  new  systems 

■  T&E  communities  need  to  understand  one  another 
first! 

■  T&E  Communities  forging  closer  working 
relationships 

■  Different  priorities 

■  “Cultural”  and  “Language”  differences 


Today's  Discussion 


■  Look  at  the  Non-Technical  issues  affecting  T&E 

■  T&E  Communities  forging  closer  working 
relationships 

■  Different  priorities 

■  “Cultural”  and  “Language”  differences 

■  Multi-Service  and  Joint  Testing 

■  OSD-driven  -  “Testing  in  a  Joint  Environment 
Roadmap” 

■  What  is  the  impact  to  Service  Acquisition  efforts? 

■  PMs,  T&E,  Warfighters  must  work  together  across 
Service  boundaries  to  field  new  systems 

■  First  Step:  T&E  Communities  must  work  together 
and  speak  the  same  language 


Non-Tecnnical  Factors 


Affecting 


The  Problem: 

How  to  clarify  Lifecycle  issues  in  a  T&E  context? 


Tactical  Use  Cas<  M&S 
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Acquisition  Experimentation 


^Joint  Capabilities  Integration 
and  Development  System  4 


Tactical  “Realities”  '-4?^- 


Threats 


Challenges 

■  Systems  of  Systems  (SOS)/Families  of  Systems 
(FOS) 

■  Network  Enabled  Systems 

■  Network  Centric  Enterprise  Services 

■  Global  Information  Grid 

■  Joint  Command  and  Control  (JC2) 

■  Multi-National  Information  Sharing 

■  Missions  and  Scenarios  Paradigm  Shift 

■  Transform  from  forces-based,  materiel-centric  Cold 
War  to  capabilities-based,  mission  centric 
asymmetric-warfare  posture 

■  Joint  Focus 


What  is  the  “force  multiplier?” 
How  do  we  define  it? 


Non-Technical  Factors 

T&E  Community  Issues 


■  “Ad  Hoc”  Processes 

■  Rely  on  individuals,  not  processes,  to  successfully 
complete  events 

■  Common  Tools 

■  Few 

■  Not  used  effectively 

■  Gap  in  tools  available  for  collaboration  and 
communication 

■  Not  familiar  with  other  Services 

■  No  effective  Network  Engineering  Process 

■  Insufficient  process  maturity  to  be  “repeatable” 


A  Cultural  Change 


■  Joint  vice  Service  focused  testing 

■  Shared  Models  and  Simulations 

■  “Users  won’t  use  the  model  correctly  and  it  will 
reflect  badly  on  me/us...” 

■  Improved  understanding  of  Joint  Test  environment 
including  Network  and  Security  strengths  and 
limitations 

■  Navy  =  Air  Force  =  Army  =  Marines 

■  Improved  community-wide  methods  and  processes 

■  JTEM  is  taking  the  first  steps 

■  Sustainable,  Repeatable,  Consistent, 
Understandable  results 

■  Operations  and  Acquisitions  context 


The  Benefit: 

Better  Info  to  Decision-Makers,  Faster 


■  Operational  and  Acquisition  Leadership 

■  Better  analysis  of  alternatives  for  acquisition 
decisions 

■  Clearer  understanding  of  test  results  across  T&E 
and  Operations  Spectrum 

■  Shared  data  and  information  -  faster  analysis  and 
recommendations 

■  Virtual  Environment  for  testing 

■  Capability-based  assessment  of  system 

■  Examination  of  proposed  systems  from  a 
Doctrine/Operations/Training  perspective  early  in 
development. 

■  Common  “language”,  data,  and  processes 
between  Acquisition,  T&E,  and  Ops  Communities 

■  Common  understanding  between  Acquisition, 
Ops,  and  T&E  Communities 


Capabilities-Based  Development 

The  Missions  and  Means  Framework _ 

■  The  LINK  between  the  Military  Decision  Making 
Process  and  the  domain  of  DOTMLPF  solutions 

■  A  WARFIGHTER-FOCUSED  STRUCTURE  for 
rigorous,  complete,  and  detailed  analysis  in 
crucial  evaluation  programs 


■  An  ORGANIZING  PRINCIPLE  for  requirements,  test 
planning,  and  evaluation 


Application  of  trie  Mission 


and  Means  FrarnewofS  to 
Distributed  Testing: 


oorne  f 


Distributed  Test  Event-5/ Multi-Service 
Distributed  Event 
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Event:  August  2005 

Mission  Context:  A  Company  Mounted  Operation  Supported 
by  Platoon  Dismounted  Forces  in  an  Urban  Environment 


Sten  1:  Alpha  Team  (2  ICV  &  1  MCS  PLTs) 
moves  along  AXIS  BLUE.  Enemy  dismount  force 
is  detected  by  A  TM's  Class  II  UAV  in  town.  ICV 
PLT  identifies  enemy  flank,  and  formulates  & 
disseminates  plan  to  all  Soldiers.  MCS  PLT  moves 
to  SBF  position.  ICV  PLT  moves  to  secure 
locations  &  deploys  Class  I  UAV,  SUGV,  and  T- 
UGS.  ICV  PLT  dismounts  and  attacks  with  direct 
fire  &  movement;  deploys  U-UGS.  ICVs  & 
weapons  squads  provide  supporting  LOS  fires. 
MCS  PLT  provides  supporting  BLOS  fires. 


Step  2:  Enemy  force  in  town  is  defeated  or 
captured.  ICV  PLT  secures  objective,  and 
consolidates  &  reorganizes.  Dismount  PLT  LDR 
transmits  situation  report  &  requests  MEDEVAC 
support.  MCS  PLT  covers  dismount  movement 
to  ICV  remount  locations.  ICV  PLT  quickly 
remounts  ICVs.  ICV  PLT  LDR  updates  SAfor  all 
Soldiers.  MCS  Class  II  UAV  detects  and 
identifies  another  dismounted  force.  Alpha  Team 
continues  attack  along  AXIS  BLUE. 


Event  Joint  Mission  Context 


AIR  FORCE 


DMOC  -  Kirtland  AFB,  NM 
SIMAF  -  Wright  Pat  AFB,  OH 
46thTW-Eglin  AFB,  FL 
Hurlburt  Field,  FL 
CAOC-X  -  Langley  AFB,  VA 
AFAMS-CVC  -  Pentagon 
SWC  -  Schriever  AFB,  CO 


NSWC  -  Navy 


NAVY 

NSWC  -  Dahlgren,  VA 
NAWCAD  -  Patuxent  River,  MD 
SPAWAR  -  San  Diego,  CA 
NAWCWD  -  China  Lake,  CA 


ARMY 

ATC  /  DTC  -  APG,  MD 

EPG  -  Ft  Lewis,  WA 

EPG  -  Ft  Huachuca,  AZ 

RTTC/AMRDEC  -  Huntsville,  AL 

WSTC  /  IRCC-  WSMR,  NM 

JPSD  JPO  /  NVESD  -  Ft  Belvoir,  VA 


WSMR  -  Army 


JITC  -  Joint 


Participants 


Event  Analysis  Framework 


Missions  and  Means  Framework 


11  Fundamental  Elements:  j  Seven  Levels,  Four  Operators 


7.  OWNFOR  Why  =  Purpose,  Mission 


Employment 


MSDE  Analysis  Framework 
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Use  Joint  Tactical  Tasks  to  define  the  operation 
Decompose  the  tasks  to  understand  component-level 
influences 

Instrument  the  components  to  measure  their  activities 
Synthesize  the  tasks  as  a  manifestation  of  component 
interactions 


Event  Analysis  Methodology  nina 


ANALYSIS  PRODUCTS 


Baseline  analysis  methodology  for 
assessment  of  Joint  Tasks 

Baseline  set  of  Joint  Task  Measures 
associated  with  each  Service  Level 
Task  (NTA,  ART,  AFT,  MCTL) 

■  Quantitative  Data  Examples 

■  ART  1.4.3  -  M3  -  Time  to  make 
initial  assessment  of  attacks 
after  TOT 

■  NTA  3.2.2  -  M2  -  Minutes  after 
target  ID  to  complete  attack 

■  AFT  2.1 .1  -  Ml  -  Time  from  the 
desired  timing  for  lethal  force 
to  cause  desired  effects 

System  level  measures  for  respective 
service  test  objectives 


MULTI-SERVICE  JOINT  TASK 
COMMON  ANALYSIS 
METHODOLOGY 


Results  and  Areas  for  Improvement  ^ 


Demonstrated  ability  to  execute  a  distributed  L/V/C  event  with  existing  capabilities. 
Conducted  system  testing  in  the  context  of  a  Joint  mission. 

Gained  experience  with  multi-Service  performance  report  generation  using  multiple  tools. 
Gained  experience  in  reporting  complex  Joint  thread  exercises  with  diverse  data  formats. 


Areas  Requiring  Improvement 

1 

Ability  to  determine  system  contributions  to  the  accomplishment  of  a  Joint  task. 

2 

Ability  to  evaluate  individual  test  item  performance  in  LA//C  events. 

3 

Ability  to  test  multiple  items  in  a  Joint  environment  simultaneously  and 
accomplish  all  test  objectives. 

4 

Ability  to  sufficiently  gather,  process,  and  analyze  data  (one  set)  from  distributed 
L/V/C  events. 

5 

Determine  capabilities  and  limitations  associated  with  legacy  systems  in  Joint 

L/V/C  events. 

6 

Integrate  across  a  variety  of  environment  models  and  coordinate  systems. 

7 

Assess  and  integrate  multiple  threat  representations  in  a  distributed  L/V/C 
environment 

Getting  Organized  to 
Perform  E'ialiiatleir, 


j^-^THu|pn  ReS€afq|&  Efigiijeering  Directorate, 
U.S.  Army  Research  Laboratory 


3/14/2006 


J 

r 

Mode  1 

L 

r  Dc 

iliben 

lannir 

ite 

Military  Operations 
and  Warfighting 


Require  Capability 


▼ 


1 


Provide  Capability 


DOTMLPF*  Solutions 


Define 


▼ 


T 


Enable 


Research, 


Experimentation 


Logistics, 

Sustainment 


Readiness, 

Training 


*  DOTMLPF:  Doctrine,  Organization,  Training,  Materiel,  Leader  Development,  Personnel,  Facilities 


J 

r  ^ 

Mode  1 

L 

r  Dc 

iliben 

lannii 

ite 

Military  Operations 
and  Warfighting 


Require  Capability 


▼ 


1 


Provide  Capability 


DOTMLPF*  Solutions 


Define 


▼ 


T 


Enable 


r 


Analysis, 

Demonstration 


Research, 
Experimentation 


Tactics, 

Fielding 


Life  Cycle 


Acquisition, 
Test  &  Evaluation 


r  i 

Logistics, 

Sustainment 

k.  A 

f 

1 

r  ^ 

Readiness, 

Training 

L, _ A 

*  DOTMLPF:  Doctrine,  Organization,  Training,  Materiel,  Leader  Development,  Personnel,  Facilities 


The  Russians  Are  Coming,  The  Russians  Are  Coming® 


"Just  got  to  get  organized.  We've  got  to  get  organized.” 


Jonathan  Winters  (Officer  Norman  Jones)  to  Ben  Blue 


§Metro-Goldwyn-Mayer,  Inc.,  1966 
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Ballistic  Live  Fire  Example  - 1985 
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Direct  Fire  Validation* 
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System  Evaluation  Metrics 
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Personnel 

Discrete  Distribution  | 
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Perforations 

Component  Criteria 

Personnel  Criteria 

Discrete  Distribution 

□  >50% 

□  >  80% 

□  <  1/2  Crew  member 

□  >50% 

O  >  5%  &  <  50% 

O  >  50%  &  <  80% 

O  >  1/2  &  <  1  crew  member 

O  >  5%  &  <  50% 

O  >  0%  &  <  5  % 

O  <  50% 

O  >  1  Crew  member 

O  >  0%  &  <  5  % 

X  =0% 

X  =0% 

*  From  William  E.  Baker,  Richard  Saucier,  Theodore  M.  Muehl,  and  Ricky  L.  Grote,  1998. 
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4.  Activity-centric,  named  with  a  Verb,  “Do  What” 

“The  Playbook” 
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6.  Context,  Environment  (Military,  Civil,  Physical,  etc.) 
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Architecture  defines  how  Parts  are  assembled  into  Packages 
Capabilities  are  relationships  between  Parts  and  Packages 
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EMRL  1  @  SoS  PDR 


Preliminary 
Design 


Evolve  Cl  Functional  Specs  into 
Product  (build  to)  Documentation 


MS  C  2012  (4Q) 


Verification 


•  Final  Industrial  Capability 
Assessment  Complete  m 

•  Production  Readiness 
Reviews  (PRR) 
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Validation 
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System 
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Verification 


System  Level  DT&E  /  LFT&E 
Verify  System  Performance 
Compliance  to  Specs 


Subsystem 

Integration 

Verification 


Integrated  DT&E  /  LFT&E 
Verify  Performance 
Compliance  to  Specs 


CI/CSCIs 


Component 

Verification 


Individual  Cl /CSCI 
Verification 


Concurre  it  Engineering  Approach 


SoS  CDR  2010(4Q) 
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Documentation 
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The  FCS  BCT  Integrates  With 
Army  Enterprise  System  Into  the 
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Integrated  Warfighting 
Platforms:  Lethality  / 


Survivability  enhanced 
underlying  network 


Sensing  Systems 

ivistisi,  atidibte,  Jtc. 
and  Ears  of  the  Commander 


BC  Applications:  assimilate  info 
BC:  C2  logic  and  reasoning  based 
on  information 


System  Services:  Common  Net- 
Centric  Infrastructure  SOSCOE: 
Tactical  Net-Centric  Middleware 


Transport  Systems:  networked 
communications 
Networked  Communications: 
comms  backbone  and  omms 
subnets  from  GIG  to/from  ground 
sensors 


a-MIND™§- "Automated  Mission 
Relevant  Situational  Awareness" 


What  Is  done? 

Operational 

Architecture/Operational 
Use  Case  Definition 


How  It's  done. 


Operational  Use  Cases 
Dependencies  on 
Services  Service  Layer  - 
Virtual  Translator 


What  Is  used? 


Infostructure  Used  to 
Provide  Services 


Operational  Architecture 


Information  Infostructure 


a-MIND 
Technology 
automates 
dependency 
understanding 
enabling  analysis 
of  Mission  Impact 
of  Infostructure 
Disruptions 


§  Mission  Impact 
Management  (MIM) 
Solution  a  Product  of 
Northrop  Grumman- 
Patent  Pending 


Proven  ability  to  integrate  COTS  products 

Unique  integration  and  analysis  framework  -  patent  filings 


n  Dependencies 

Map  and  view  relationships  within  tiers  . . . 


Dependencies 

. . .  and  between  tiers 


A-MIND  -  [  Map-2]  [T]@® 


^  File  Edit  View  Tools  Window  Help  _  &  X 


Functional  Performance 


New  M&S  Requirements  Module 


Common  Critical  CI/CSCIs 
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Network  Systems 

Communications 

Interoperability 

Networked  Lethality 

Training 

Survivability 

Maneuver  /  Maneuver  Support 

MANPRINT 

Sustainment 

Transportability/Deployability 

System  Management 

RAM-T 

Constraints  &  Environments 

Safety  (ESOH) 

Producibility 

Affordability 

Growth 

Modeling  and  Simulation 

|  | 


Requirements  Modules 
Established  Around  the 
Front-Face  of  “the  Cube” 


M&S  Module  is  intended  to 
enhance  the  integration  and 
utilization  of  M&S  across  all 
aspects  of  the  program. 


■  M&S  Team  is  responsible 
for  the  application  of  M&S 
from  “Cradle-to-Grave”  and 
“Top-to-Bottom” 


M&S  Requirements  Module 

"□0 


Multi-disciplined  Teams  including  LSI,  PM  UA  &  TRADOC 


Simple  Tree  —  UGV  Example 


Conclusions 


■  T&E  and  M&S  have  evolved  and  are  managed 
asynchronously 

■  Adequate  evaluation  for  today’s  complex  SoSs  in  a 
Joint  context  requires  extensive  complimentary 
Test/M&S  detailed  planning  and  execution 

■  Lack  of  common  processes  inhibits  M&S  sharing  in 
the  T&E  community 

■  No  standard  “Standard”  to  assess  M&S  ability 

■  What  is  a  “High  Fidelity  Model”? 

■  No  common  processes  for  using/sharing  M&S 

■  Common  understanding  through  common 
languages,  methods,  and  processes  are  crucial  to 
achieving  full  T&E  community  integration 


Path  Ahead 


■  Crawl  before  Walking 

■  Develop  a  common  understanding  of  what 
“Warfighter  focused  T&E”  means  to  each  Service 
and  DoD  Agency 

■  Develop  common  language  and  practices  for  M&S 
use  in  T&E 

■  Understand  Service/Agency  strengths  and 
weaknesses  in  M&S  in  a  T&E  context 

■  Community-wide  Methods  and  Processes  (M&P) 

■  Standard  M&P  for  T&E  across  all  Services  and  DoD 
Agencies 

■  Links  M&S  use  within  DoD  T&E  communities 

■  Good  news:  JTEM  is  making  progress  in  this  area 


Path  Ahead 


■  Incorporate  MMF  as  a  community-wide  T&E 
Process 

■  Global  organizing  schema  covering  both  the  mission 
definition  and  mission  execution 

■  Common  understanding  of  T&E  in  Warfighter 
context 

■  Without  a  global  organizing  schema  covering  both 
the  mission  definition  and  mission  execution  sides  of 
the  problem,  the  many  pieces  cannot  be  properly 
defined,  instantiated,  linked  and  executed 
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How  to  clarify  Lifecycle  issues  in  a  T&E  context? 
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Points  of  Contact 


■  John  lllgen,  Northrop  Grumman  SIM  TECH 

■  Phone:  (805)692-2333X201/206 

■  john.illgen@ngc.com 

■  Dr.  Paul  H.  Deitz,  Director(A),  U.S.  Army  Research  Laboratory 

Human  Research  &  Engineering  Directorate 

■  Office:  (410)278-5800  Cell:  (443)421-0039 

■  paul.h.deitz@us.army.mil 

■  Mr.  Rick  Cozby,  Chief,  Technology  Management  Division 
HQ,  U.S.  Army  Developmental  Test  Command 

■  Phone:(410)278-1474  DSN:  298-1474 

■  rick.cozby@dtc.army.mil 

■  Mr.  Jack  H.  Sheehan,  Chief  Engineer  Combined  Test 

Organization,  PM  Unit  of  Action 

■  Office:  (703)647-1448  Cell:  (443)831-2385 

■  jack.h.sheehan@fcscto.army.mil 

■  Augustine  Ponturiero,  Northrop  Grumman  SIM  TECH 

-  Phone:  (240)682-1886 
"  augustine.ponturiero@ngc.com 
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execution  and  adjudication  of  outcomes 
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Architecture  defines  how  Parts  are  assembled  into  Packages 
Capabilities  are  relationships  between  Parts  and  Packages 
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Jacksonville,  FL 
March  8,  2006 

Good  afternoon.  Like  many  of  you,  for  years  I’ve  followed  the 
advances  of  modeling  and  simulation  and  listened  to  promises  of  cost 
savings  and  better,  faster,  more  efficient  testing.  And  I’ve  always 
been  a  huge  fan  of  M&S,  but  the  reality  didn’t  always  live  up  to  the 
promise.  In  the  early  days  the  M&S  folks  would  say,  “We  are  10-15 
times  cheaper  than  open  air  testing.”  But  when  you  added  in  the  cost 
of  the  simulation  facilities,  the  cost  to  develop  the  simulations,  and 
the  cost  of  open  air  testing  just  to  validate  the  simulation,  it  was 
sometimes  hard  to  find  any  savings  at  all. 

And  later  as  our  systems  got  more  complicated  the  M&S  folks  would 
say  “You  need  us  now  for  repeatability  and  accuracy  and  speed,  and 
to  help  you  determine  where  to  look  during  open  air  testing,  and  to  do 
all  the  software  ‘what  if’s’,  and  we  are  10  to  15  times  cheaper.”  All 
true  but  not  compelling  because  when  it  came  down  to  it,  there  was 
usually  some  glitch  or  issue  that  affected  the  accuracy  of  the 
simulation.  So,  like  many  of  you,  I’ve  been  disappointed  when  the 
reality  didn’t  live  up  to  the  hype.  A  retired  Admiral  friend  of  mine  once 
compared  M&S  to  the  economy  of  Brazil  -  it  has  wonderful  potential, 
and  always  will  have! 


As  a  result,  M&S  has  been  the  step-child  of  RDT&E.  And  while  there 
have  been  some  great  success  stories,  what  we  remember  most  are 
the  stories  of  mistakes  and  foibles.  One  of  my  favorites  involves  our 
allies  the  Aussies,  who  I  had  the  pleasure  to  work  with  in  Melbourne 
from  1989-  1992. 

As  virtual  reality  simulators  assumed  larger  roles  in  combat  training, 
programmers  went  to  great  lengths  to  increase  the  realism  of  their 
scenarios,  including  detailed  landscapes  and--in  the  case  of  the 
Australian  Northern  Territory  -  they  included  herds  of  kangaroos  and 
then  modeled  the  local  marsupials'  movements  and  their  reactions  to 
low  flying  helicopters  (since  disturbed  animals  might  well  give  away  a 
helicopter's  position). 

Being  efficient  programmers,  they  just  re-appropriated  some  code 
originally  used  to  model  infantry  detachment  reactions  under  the 
same  stimuli,  changed  the  mapped  icon  from  a  soldier  to  a  kangaroo, 
and  increased  the  figures'  speed  of  movement. 

Eager  to  demonstrate  their  simulator  for  some  visitors,  the  hotshot 
Aussies  "buzzed"  the  virtual  kangaroos  in  low  flight.  The  kangaroos 
scattered,  as  predicted,  and  the  visitors  nodded  appreciatively...  then 
did  a  double-take  as  the  kangaroos  reappeared  from  behind  a  hill  and 
launched  a  barrage  of  Stinger  missiles  at  the  helicopter. 

And,  in  fact,  our  M&S  and  software  folks  have  been  the  brunt  of 
several  great  jokes.  My  personal  favorite  is  the  one  about  the 


Program  Manager,  chief  engineer,  head  of  logistics,  and  M&S  lead 
who  are  riding  together  in  a  car  that  has  suddenly  lost  its  breaks  and 
is  careening  down  a  mountain  road.  Fortunately,  the  road  followed 
an  old  logging  trail  with  several  “runaway  trunk  ramps”  built  in.  Just 
when  it  looks  hopeless,  the  Program  Manager,  who  is  driving,  sees 
one  of  the  old  runaway  truck  ramps  and  steers  into  it  to  stop  the  car. 
The  PMA  immediately  says,  "I  want  a  tiger  team  to  investigate  this 
incident,  want  a  POA&M  to  address  our  recovery  plan,  we  need 
funding  established  and  all  POCs  indentified."  The  Chief  Engineer 
says,"  I  want  to  field  strip  these  brakes,  do  an  non-destructive 
investigation,  stress  analysis  and  fault  tree  analysis  and  get  the 
brakes  back  in  service  by  tonight!  The  Chief  Logistition  says,  “I  want 
to  see  which  parts  are  non  RFI,  do  an  RCM  analysis  on  the  brakes, 
and  have  the  new  parts  ready  to  install  by  early  this  afternoon.  The 
M&S  lead  says,  "Hmmm,  I  didn’t  see  this  in  the  simulation,  let's  push 
the  car  back  up  to  the  top  of  the  mountain  and  see  if  it  does  it  again!" 

Well  I  think  we  did  ourselves  a  disservice  by  pinning  the  benefits  of 
m&s  on  speed,  accuracy,  repeatability  and  cost  savings.  Yes,  these 
are  important;  but  if  this  is  your  only  rationale,  you  face  the  value 
issue  --  while  the  cost  might  be  less,  the  product  is  worth  less.  As  all 
of  us  who  have  been  involved  in  test  know,  there  is  no  substitute  for 
open  air  testing. 

But  in  the  last  few  years,  we’ve  seen  a  growing  tolerance  for  M&S. 
There  are  places  where  “open  air  testing”  doesn’t  want  to  go  or  can’t 


easily  go  and  is  begrudgingly  happy  to  have  M&S  around  to  help. 
Where  might  some  of  those  places  be? 

First,  dangerous  tests  -  live  fire  testing  might  count  here  --  where  the 
test  article  and  the  projectile  are  real  but  the  environment  is 
simulated.  But  if  you  don’t  want  to  count  that  as  a  true  simulation, 
how  about  testing  warning  systems  --  like  aircraft  ground  proximity 
warning  systems,  or  collision  avoidance  systems  where  you  want  to 
point  the  aircraft  at  the  ground  or  another  aircraft  and  see  if  the 
automatic  warning  comes  on  just  in  time  to  avoid  disaster. 

Second,  there  are  tests  where  security  could  be  compromised  if  done 
in  open  air  such  as  testing  an  emitter  at  a  new  frequency,  of  testing 
the  war  modes  of  a  system,  or  trying  new  tactics  during  OT  that 
spying  eyes  might  see. 

Third,  there  are  tests  where  it’s  impractical  to  do  open  air  testing  - 
jamming  in  open  air  -  we  once  knocked  out  all  the  TV’s  on  the  east 
coast  testing  a  new  EA-6B  jamming  pod;  or  because  of  the  density  of 
the  threat  environment  can’t  be  replicated  at  one  of  our  electronic 
ranges.  We  did  the  V-22  Operational  Assessment  on  threat  detection 
capability  and  response  in  the  Air  Combat  Environment  Test  and 
Evaluation  Facility  at  Pax  River.  By  the  way,  it  was  critical  to  that  test 
was  that  the  Operational  Testers  had  certified  that  for  the  purposes  of 
the  test,  the  facility  was  operational  representative. 


And  finally  there  are  tests  that  are  just  too  big  to  routinely  be  done 
with  open  air  testing.  As  our  systems  get  more  complicated  and 
integrated  together,  it’s  hard  to  coordinate  live  assets  for  test  so  we 
let  M&S  go  first  to  work  out  any  kinks.  Testing  MIDs  (Multi-Functional 
Information  Distribution  System)  which  is  a  new  system  for  sharing 
data-link  and  voice  in  the  battle  field,  required  numerous  ships  and 
aircraft  operating  together  to  test  the  ability  of  the  system  to  provide  a 
flow  of  information.  The  first  time  we  tested  MIDs  on  a  large  scale 
was  in  simulation  to  ensure  we  had  the  systems  working  OK  before 
tying  up  a  large  number  of  real  battle  group  assets.  And  sometimes 
the  new  systems  or  the  systems  with  which  our  system  under  test 
must  interface  are  also  in  development  and  aren’t  yet  available  and 
we  have  no  choice  but  to  use  modeling  and  simulation. 

As  our  systems  get  more  complex,  we  can  expect  that  we’ll  need 
M&S  more  and  more  in  the  future.  And  our  systems  will  get  more 
complex.  There  is  a  saying,  “As  long  as  all  things  are  created  from 
scratch,  growth  can  at  most  be  linear”.  We’re  too  clever  to  create 
things  from  scratch  so  you  can  expect  that  we  will  continue  to  use  a 
building  block  approach.  Accordingly  the  complexity  of  our  systems 
will  continue  to  increase  non-linearly  and  modeling  and  simulations 
will  become  more  and  more  important  to  prove  that  they  are 
interoperable. 

In  fact,  in  June  2003,  DoD  put  out  a  memo  that  declared  their 
intention  to  test  for  interoperability  over  the  life  of  the  system. 


So  I  think  all  of  us  involved  in  research,  development,  test,  and 
evaluation  believe  that  M&S  finally  has  a  seat  on  the  bus.  It  may  not 
be  a  window  seat,  but  it’s  a  seat.  For  RDT&E,  M&S  is  tolerated, 
thank  you  very  much. 

In  fact,  if  M&S  was  represented  by  a  character  in  a  movie,  it  would  be 
the  hapless  city  slicker  in  a  Western  --  trying  to  start  a  new  life  in  the 
rough  and  tumble  prairie.  I  can  imagine  a  scenario  where  the 
tenderfoot  is  proudly  standing  on  the  front  porch  of  his  newly 
purchased  ranch  house,  wearing  his  brand-new  cowboy  boots, 
surveying  his  vast  holding.  He  is  approached  by  a  rough  and  tumble 
cowboy  who  represents  “Open  Air  Testing”.  The  cowboy  says  to  the 
tenderfoot.  “Welcome  neighbor.  I  live  about  5  miles  down  the  road.  I 
want  to  have  a  little  get  together  tonight  to  welcome  you  to  the 
territory.” 

“Oh,  that  would  be  great”  says  the  Tenderfoot. 

“Yeah”  says  the  old  cowboy,  come  on  over  about  sunset,  we’ll  do 
some  eating  and  drinking.”  “some  fiddling  and  some  dancing” 
“probably  be  some  shooting”  “definitely  be  some  fighting”  “but  they’ll 
be  some  loving,  too”. 

“Who  all  is  coming?”  asks  the  Tenderfoot 


Oh,  just  the  two  of  us.”  Says  the  cowboy 


As  our  ability  to  accurately  model  systems  increases,  and  as  the 
memory  and  processing  power  of  our  computers  increases,  it’s 
natural  that  our  RDT&E  infrastructure  would  support  new  areas  in  the 
life-cycle  of  our  systems.  Two  of  these  -  demonstration  and 
experimentation  --  are  at  the  beginning  of  a  system’s  life,  and  two  of 
these  -  training  and  rehearsal  -  are  later  in  the  system’s  life.  The 
infrastructure  built  to  support  research,  development,  test  and 
evaluation  -  the  open  air  ranges,  software  support  activities, 
hardware  in  the  loop  facilities  and  our  facilities  that  create  synthetic 
environments  -  is  becoming  host  to  rehearsal,  demonstration, 
training,  and  experimentation  -  or  the  new  RDT&E. 

In  fact,  some  months  10%  of  the  events  we  support  with  our 
infrastructure  are  either  rehearsals,  demonstrations,  training 
evolutions,  or  experimentation  --  and  it  is  growing.  In  the  manned 
flight  section  of  ACETEF,  this  percentage  has  grown  from  50%  to 
80%  in  the  last  few  years.  Nearly  every  test  flight  is  rehearsed  in  the 
ACETEF  before  it’s  flown.  And  in  this  new  RDT&E,  modeling  and 
simulation  doesn’t  just  have  a  seat  on  the  bus,  it  is  driving  the  bus. 

In  our  business,  if  you  want  to  know  what’s  important  and  what  folks 
are  willing  to  support,  you  follow  the  money.  Joint  Forces  Command 
is  competing  a  five  year  program  starting  in  Sep  to  support 
USJFCOM’s  Joint  Trainer  responsibilities  for  planning  and  conducting 


exercises,  crisis  rehearsals  and  other  training  events  from  inception 
through  execution. 

The  long-term  mission  of  this  initiative  is  to  incorporate  service 
branches,  interagency  and  multinational  coalition  partners.  By  2009, 
the  goal  is  to  have  the  capability  to  train  any  audience  in  joint 
warfighting  -  whether  it  be  a  unified  command  staff,  one  of  the 
services,  or  multinational  and  interagency  personnel.  The  persistent 
network  will  focus  on  joint  training,  experimentation,  testing, 
education  and  mission  rehearsal,  by  linking  command  and  control, 
training  facilities,  and  ranges  and  simulation  centers  throughout  the 
world. 

Meanwhile  DoD  is  creating  the  Joint  Mission  Environment  Test 
Capability  to  provide  testers  and  developers  a  robust  nation-wide 
distributed  engineering  capability,  by  integrating  live-virtual- 
constructive  simulations  with  systems  under  test,  giving  the 
Department  the  capability  to  “Test  like  we  fight.”  DoD  wants  to 
ensure  interoperability  between  JMETC  and  JNTC,  and  is  developing 
the  JMETC  to  support  the  JNTC. 

Here’s  what’s  interesting  to  me.  A  member  of  the  new  RDT&E  -  in 
this  case  training  -  is  setting  the  standard  for  traditional  RDT&E  -  in 
this  case  testing.  We’re  likely  to  see  more  of  this  in  the  future. 

And  oh  yes,  M&S  does  save  money  in  the  new  RDT&E  --  especially 
with  training.  Real,  measurable,  auditable,  money.  All  the  services 


have  accepted  simulation  as  a  replacement  for  flight  time  -  not  just 
an  augmenter  for  flight  time.  So  have  our  allies  -  at  the  French  Army 
Aviation  Training  Centre  30%  of  the  training  is  performed  in 
simulators  and  procedural  trainers.  There  are  lots  of  examples  from 
the  commercial  side.  Most  new  airline  pilots  do  100%  of  their  training 
in  simulators  -  the  first  actual  flight  is  with  passengers! 

Very  light  jets  are  preparing  to  make  their  debut  this  year.  The 
Eclipse  500  is  tentatively  scheduled  for  FAA  approval  this  month. 
These  jets  cost  under  $3M  each  and  are  powered  by  2000  lb  thrust 
engines  and  weigh  less  than  12,500  lbs.  Now  you  can  always  get 
insurance,  especially  since  many  of  the  inexperienced  folks  who  can 
afford  these  personal  jets  can  afford  the  insurance.  But  the  key  to 
getting  affordable  insurance,  according  to  the  under-writers,  will  be  to 
participate  in  a  highly  focused  and  type-specific  training  program  with 
significant  simulation.  A  private  pilot  with  a  multi-engine  rating  and 
just  500  hrs  total  time  who  completes  the  manufacturers  training 
program  would  pay  about  the  same  amount  for  insurance  as  a 
commercial  pilot  with  2500  hours  and  500  hours  of  multi-engine  time 
would  pay. 

A  growing  use  of  simulators  is  in  a  field  related  to  training  -  mission 
rehearsal.  Complicated  missions  are  being  flown  in  rehearsal  before 
being  flown  for  real.  Simulation  shows  the  pilot  the  expected  outside 
environment  (visual,  radar  and  FUR),  the  expected  defensive  threat 
laydown,  and  allows  for  rehearsal  of  the  weapons  employment. 


SOCOM  is  developing  a  Common  Environment/Common  Database 
which  will  be  integrated  into  the  simulators  for  the  US  Army  Special 
Operations  Regiment.  They  have  compiled  a  wide  range  of  imagery, 
navigation,  communications  and  radar  data  into  a  centralized 
database.  Members  of  the  Regiment  are  able  to  create  customized 
simulation  of  the  actual  combat  zone  where  they  will  be  deploying  an 
do  it  in  hours,  not  months.  One  of  the  main  reasons  for  the 
development  is  to  shorten  the  mission  rehearsal  timeline.  Now 
several  of  the  other  COCOMs  are  asking  for  simulators  in  theater  - 
not  only  for  training  but  for  rehearsal  as  well. 

All  the  services  are  investigating  how  game  simulation  can  replicate 
real  operations  to  help  train  warfighters.  Everyone  realizes  that 
gaming  isn’t  a  valid  substitute  for  real-world  military  training  and  live- 
fire  drills,  but  gaming  is  being  investigated  as  accepted  as  an 
alternate  training  method.  The  industry  believes  that  videogame 
technology  can  fulfill  many  training  needs  at  a  much  lower  cost  that 
traditional  large-scale  simulations.  By  the  way,  that  may  be  the  first 
time  many  of  us  have  heard  the  words  “traditional”  and  “simulation” 
used  together. 

The  Navy  is  investigating  whether  a  video  game  that  replicates 
operations  aboard  an  aircraft  carrier  can  help  train  ship  and  aviation 
flight  deck  personnel.  “24Blue”  was  modeled  upon  flight  deck 
operations  on  the  aircraft  carrier  USS  Harry  S.  Truman.  The 
objective  of  the  game  is  to  launch  a  sequence  of  fixed-wing  aircraft — 
including  the  F-18  Hornet,  the  EA-6B  Prowler  and  the  S-3B  Viking — 
off  the  ship  before  it’s  attacked. 


Not  sure  about  gaming?  You  think  the  folks  who  built  games  aren’t 
interested  in  DoD  as  a  customer?  Consider  this:  game  development 
studios  are  the  closest  thing  we  have  to  steady  simulation  talent  - 
and  Ben  Sawyer  a  gaming  expert  and  founder  of  the  “Serious  Games 
Summit”  which  was  held  in  Arlington,  VA  in  2005,  claims  that  most 
successful  gaming  firms  are  hybrid  studios  that  do  gaming  and 
“serious  gaming”.  Many  believe  that  the  importance  of  gaming  will 
continue  to  grow,  and  although  it  has  taken  over  the  role  of  step-child 
from  M&S,  will  one  day  it  will  have  a  role  as  important  as  M&S  does 
today. 

So  while  M&S  is  clearly  supporting  training  and  rehearsals,  what’s  it 
doing  for  demonstration  and  experimentation?  Keep  you  eye  on  this! 
Clearly  M&S  will  have  a  huge  role  as  we  look  at  the  “art  of  the 
possible”  and  develop  new  tactics  for  systems  before  they  are  even 
developed.  M&S  has  been  involved  in  past  experiments,  but  it’s  been 
awkward.  The  folks  responsible  for  the  experiments  didn’t  have 
funds  to  pay  for  the  participation  of  new  systems  and  had  to  have 
program  managers  and  others  offer  their  support.  Today,  the  value 
of  experiments  is  clear  and  money  is  being  made  available  to  the 
experiment  leads  to  pay  for  the  systems  they  believe  are  needed. 

So  what  is  the  future?  I  believe  the  new  RDT&E  will  drive  M&S  in  the 
future.  And  while  there  are  several  600  lb  gorillas  in  DoD  that  will 
define  M&S  interface  standards  (JSF,  FCS,  JTRS)  most  of  the 
standards  for  M&S  will  come  from  self-forming  groups  of  interested 


folks  who  reach  agreement  on  where  we  need  to  go.  And  this  group 
better  include  our  contractor  teammates. 

So  the  future  for  M&S  is  pretty  good.  Now,  let’s  go  back  to  our 
tenderfoot  standing  on  his  porch  and  the  grizzled  cowboy  that  was 
having  a  little  fun  at  the  his  expense.  As  the  cowboy  starts  to  leave 
he  says  with  a  grin,  “See  you  tonight,  pardner.  And,  oh  by  the  way, 
my  name’s  Tex.  What’s  your’s?”  “Wayne”  says  the  stranger,  “John 
Wayne.”  Yes  the  future  will  be  very  interesting  indeed.  Thank  you. 
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We  are  THE  path  to  success  for  delivering  warfighting  capabilities 


Overview 


U.S.AIR  FORCE 


■  Background 

■  Need  for  “Teaming” 

■  Natural  Tensions 

■  What  Can  Be  Done  to  Team  Together 

■  Examples  of  AFFTC  Teaming 

■  Recommendations 


We  are  THE  path  to  success  for  delivering  warfighting  capabilities 


U  S- AIR  FORCE 


Background 


■  Role  of  DT&E — independent  verification 

■  Does  system  do  what  it  was  designed  to  do? 

■  Role  of  OT&E — independent  validation 

■  Does  system  do  what  it  is  supposed  to  do? 

■  Role  of  M&S  in  DT&E 

■  Initial  design 

■  Refine  design 

■  Test  design  when  conventional  means  impractical/not 
feasible 

■  Role  of  M&S  in  OT&E 

■  Test  system  when  conventional  means  impractical/not 
feasible 


We  are  THE  path  to  success  for  delivering  warfighting  capabilities 
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Natural  Tensions  Between 
DT&E  and  M&S  Communities 


DT&E  may  feel  threatened  by  M&S 
■Less  Testing 


Less  DT  Work 


Less  Job  Security 


■PM  may  feel  DT&E  too  expensive... takes  too  long 

■M&S  says,  “We  can  do  it  all!!” 

Possible  Implication: 

PM  question  value  of  DT&E  and  be  motivated  to 
perform  testing  via  M&S  only 


•Global  Hawk 


WRONG  APPROACH 


We  are  THE  path  to  success  for  delivering  warfighting  capabilities 


Current  DT&E/M&S  Interaction 


U.S.AIR  FORCE 


DT&E/M&S  Independent  Relationship 


High  Risk  Tests 

Envelope 


Routine  Tests 


Within  Envelope 
Testing 

”  ”  ”  ~  y 

Unclear 

relationship 


High  Risk  & 

Routine  Tests 

Envelope  Expansion 
/  Within  Envelope ^  ^ 

A_  _  Y 

s _ M  DT&E 

^  \ 

Unclear  ^ 

relationship 


Preferred 

Relationship 


Envelope  Expansion 
/  Within  Envelope 


Relationship 


We  are  THE  path  to  success  for  delivering  warfighting  capabilities 
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U  S- AIR  FORCE 


From  Mutual  Exclusiveness  to 
Interdependency 


■  Implications  of  Interdependent  DT&E/M&S 
Relationship 

■  Less  Testing  Needed... Less  Time  Needed... Less  Dollars 
Needed.... 

■  Better  Reputation... More  Customers.... 

■  Increase  work  for  DT 

■  Ability  to  test  more  test  articles  (increase  throughput) 

■  Higher  quality  product 


■  NCO/W  natural  “marriage”  between  DT&E  and  M&S 

■  Can’t  do  it  all  with  M&S  or  DT&E  alone 

■  Full  spectrum  of  LVC 


We  are  THE  path  to  success  for  delivering  warfighting  capabilities 
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U  S- AIR  FORCE 


How  AFFTC  connects  DT  and 

M&S 


■  AFFTC  has  successfully  teamed  with  Boeing  via  CRADA  to: 

■  Cross  government  and  contractor  networks 

■  Tied  both  M&S  networks  together 

■  Network  connectivity  enabled  the  AFFTC  and  Boeing  to  conduct 
exercises  using  Hardware  in  the  Loop  simulators 

■  AFFTC -F-16IFAST  simulator 

■  Boeing  -  F-15,  F-18,  AWACS  4045  (simulators) 

■  Also  have  used  other  Link  16  assets  in  demonstrations 

■  AFFTC  Network  Mission  Support  System  (in  particular  the  Air 
Defense  System  Integrator  (ADSI)) 

■  Link  16  Vans 

■  AFFTC  also  partnering  with  Boeing  in  JEFX  06  through  AFFTC 
DREN  link 


We  are  THE  path  to  success  for  delivering  warfighting  capabilities 
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M&S  Infrastructure  Integration  Demo 


-  June  2005 


'’Jt 


Boeing  Palmdale/ AFFTC  Edwards  AFB  & 


U.S.AIR  FORCE 


Edwards  AFB 


We  are  THE  path  to  success  for  delivering  warfighting  capabilities 
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w 


Live  &  Virtual  Data  Link 

Integration  Demonstration  —  December  2005 

Boeing  Palmdale/ AFFTC  Edwards  AFB  =_$ 


4XF-16 
(Live,  airborne) 
Edwards  AFB 


We  are  THE  path  to  success  for  delivering  warfighting  capabilities 
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4  4  JEFX06 

Virtual  Warfighter  M&S  Infrastructure  Activity 

April  2006 

r  ~  ~  Boeing  Palmdale/ AFFTC  Edwards  AFB  — 


U.S.AIR  FORCE 


We  are  THE  path  to  success  for  delivering  warfighting  capabilities 


10 


Recommendations 


U.S.AIR  FORCE 


■  Develop  Ways  for  better  DT/M&S  Partnership 

■  Look  at  how  DT&E  uses  M&S  data  in  testing  today 

■  Is  M&S  used  to  help  determine  how  much  testing  is  enough? 

■  Are  M&S  data  used  to  do  better  DT? 

■  Look  at  how  DT&E  community  feeds  test  data  to  M&S  community 
and  vice  versa 

■  Are  test  data  used  to  refine  M&S  Models? 

■  Look  at  how  DT/M&S  community  is  working  together  in 
NCO/W 

■  Are  there  lessons  learned  that  can  apply  to  DT&E/M&S  communities 
elsewhere? 


We  are  THE  path  to  success  for  delivering  warfighting  capabilities 
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How  to  Improve  the  Effectiveness,  Efficiency, 
and  Integration  of  Test  &  Evaluation  (T&E) 
and  Modeling  and  Simulation  (M&S) 


Dr.  Mark  J.  Kiemele 
Air  Academy  Associates 
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NDIA  Conference  on  T&E  and  M&S 
Jacksonville,  FL 
March  6,  2006 


Agenda 


Some  Basic  Definitions 

The  Heart  and  Soul  of  Multivariate  Testing 
Integrating  T&E  with  M&S 

Examples  of  Iterative  Use  of  Modeling  and  Simulation 
Summary  of  “Modeling  the  Simulator” 
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Some  Basic  Definitions 


System:  a  collection  of  entities  which  act  and  interact 
together  to  achieve  some  goal 


Model:  a  simplified  representation  of  a  system 
developed  for  the  purpose  of  studying  a  system 


Simulation:  the  manipulation  of  a  model  in  such  a  way 
that  it  allows  the  investigation  of  the  performance  of  a 
system. 


Air 
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Modeling  and  Simulation:  a  discipline  for  developing 
a  level  of  understanding  of  the  interaction  of  the  parts  of 
a  system,  and  of  the  system  as  a  whole 
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- 

About  Models 

> 

All  models  are  simplifications  of  reality. 

A 

There  is  always  a  tradeoff  as  to  what  level  of  detail  should  be 
included  in  the  model: 

\ 

If  too  little  detail,  there  is  a  risk  of  missing  relevant 
interactions  and  the  resultant  model  does  not  promote 
understanding 

\ 

\ 

If  too  much  detail,  there  is  a  risk  of  overlv  complicating  the 
model  and  actually  preclude  the  development  of 
understanding 

Air 

Academy 

Associates 

The  goodness  of  a  model  depends  on  the  extent  to  which  it 
promotes  understanding 

Copyright 

2006 
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Types  of  Models 


High-Fidelity  Models: 

many  variables  and  many  interactions 
highly  detailed  and  complex 
needed  for  visualization 
difficult  to  manipulate 


Low-Fidelity  Models: 

much  fewer  number  of  variables 
can  be  manipulated  more  easily 
provides  higher-level  view  of  system 
presents  a  more  aggregate  view  of  the  system 


Air 
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Types  of  Simulation 


Deterministic  Simulation: 

for  each  combination  of  inputs  parameters, 
there  is  one  and  only  one  output  value 

-  y  =  f(x) 


Monte  Carlo  Simulation: 

provides  for  variability  in  the  inputs 
y  =  f(x  +  variation),  where  the  variation  is 
modeled  as  some  probability  distribution 


Air 
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Discrete  Event  Simulation: 

studies  a  sequence  of  countable  events 
assumption  is  that  nothing  of  importance  takes 
place  between  events 


Copyright 
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Examples  of  Low  Fidelity  Models 
(Transfer  Functions) 


► 
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•  Engineering  Relationships 
-  V  =  IR 
F  =  ma 


Ri 


A/V\A 

R2 


The  equation  for  the  impedance  (Z) 
through  this  circuit  is  defined  by: 

Z=  R1  •  R2 
R1  +  R2 

The  equation  for  magnetic  force  at  a  distance 
X  from  the  center  of  a  solenoid  is: 

.5X+x  .5^-x 

^/r2  +(.5A,+  x)2  ^/r2  +(.5A,-x)2 


Where 


N :  total  number  of  turns  of  wire  in  the  solenoid 
I  :  current  in  the  wire,  in  amperes 
r :  radius  of  helix  (solenoid),  in  cm 
X'.  length  of  the  helix  (solenoid),  in  cm 
x  :  distance  from  center  of  helix  (solenoid),  in  cm 
H :  magnetizing  force,  in  amperes  per  centimeter 
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Examples  of  High  Fidelity  Simulation  Models 


► 
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Mechanical  motion:  Multibody  kinetics  and  dynamics 
ADAMS® 

DADS 


Pro  MECHANICA 

ABAQUS®  Standard  and  Explicit 

ADINA 


Implicit  Finite  Element  Analysis:  Linear  and  nonlinear 
statics,  dynamic  response 

MSC.Nastran™,  MSC.Marc™ 

ANSYS® 


Explicit  Finite  Element  Analysis  :  Impact  simulation, 
metal  forming 
LS-DYNA 
RADIOSS 

PAM-CRASH®,  PAM-STAMP 


General  Computational  Fluid  Dynamics:  Internal  and 
external  flow  simulation 
STAR-CD 
CFX-4,  CFX-5 
FLUENT®,  FIDAP™ 

PowerFLOW® 

Dr.  Srinivas  Kodiyalam,  SGI 


Examples  of  High  Fidelity  Simulation  Models 


Preprocessing:  Finite  Element  Analysis  and 
Computational  Fluid  Dynamics  mesh  generation 

ICEM-CFD 

Gridgen 

Altair®  HyperMesh® 
l-deas® 

MSC.Patran 

TrueGrid® 

GridPro 

FEMB 

ANSA 


\ 
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Postprocessing:  Finite  Element  Analysis  and 
Computational  Fluid  Dynamics  results  visualization 
Altair®  HyperMesh® 
l-deas 
MSC.Patran 
FEMB 
EnSight 
FIELDVIEW 

ICEM  CFD  Visual3  2.0  (PVS) 

COVISE 


Dr.  Srinivas  Kodiyalam,  SGI 


8 


Applications  of  Modeling  and  Simulation 


► 


Air 
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Power 


Electronics 


Simulation  of  underhood  thermal  cooling  for  decrease 
in  engine  space  and  increase  in  cabin  space  and  comfort 


Evaluation  of  dual  bird-strike  on  aircraft  engine 
nacelle  for  turbine  blade  containment  studies 


Simulation  of  stress  and  vibrations  of  turbine 
assembly  for  use  in  nuclear  power  generation 


Automotive 


Evaluation  of  cooling  air  flow  behavior 
inside  a  computer  system  chassis 


©  Dr.  Srinivas  Kodiyalam,  SGI 
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Reasons  for  Test  &  Evaluation 

> 

Functional  Compatibility: 

to  determine  if  various  components  or 
subassemblies  work  together 

\ 

Screening: 

to  separate  the  critical  parameters  from  those 
that  are  not  critical  with  regard  to  functionality 

v 

or  performance  capability 

\ 

Modeling: 

\ 

Air 

to  build  prediction  capability  of  the 
performance  measures  and  perform 
sensitivity  and  interaction  analyses  on  the 
critical  parameters 

Academy 

Associates 

Copyright 

2006 

10 

\ 


Air 

Academy 

Associates 

Copyright 

2006 


Introduction  to  High 
Throughput  Testing  (HTT) 

A  recently  developed  technique  based  on  combinatorics 

Used  to  test  myriad  combinations  of  many  factors  (typically  qualitative) 
where  the  factors  could  have  many  levels 

Uses  a  minimum  number  of  runs  or  combinations  to  do  this 

Software  (e.g.,  ProTest)  is  needed  to  select  the  minimal  subset  of  all 
possible  combinations  to  be  tested  so  that  all  n-way  combinations  are  tested. 

HTT  is  not  a  DOE  technique,  although  the  terminology  is  similar 

A  run  or  row  in  an  HTT  matrix  is,  like  DOE,  a  combination  of  different  factor 
levels  which,  after  being  tested,  will  result  in  a  successful  or  failed  run 

HTT  has  its  origins  in  the  pharmaceutical  business  where  in  drug  discovery 
many  chemical  compounds  are  combined  together  (combinatorial  chemistry) 
at  many  different  strengths  to  try  to  produce  a  reaction. 

Other  industries  are  now  using  HTT,  e.g.,  software  testing,  materials 
discovery,  IT  (see  IT  example  on  next  page) 
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HTT  Example 


► 
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An  IT  function  in  a  company  wanted  to  test  all  2-way  combinations  of  a 
variety  of  computer  configuration-related  options  or  levels  to  see  if  they 
would  function  properly  together. 

Here  are  the  factors  with  each  of  their  options: 


Motherboards  (5) 
RAM  (3) 

BIOS  (3) 

CD  (3) 

Monitor  (5) 
Printer  (3) 
Voltage  (2) 
Resolution  (2) 


Gateway,  ASUS,  Micronics,  Dell,  Compaq 
128  MB,  256  MB,  512  MB 
Dell,  Award,  Generic 
Generic,  Teac,  Sony 
ViewSonic,  Sony,  KDS,  NEC,  Generic 
HP,  Lexmark,  Cannon 
220,  110 

800x600,  1024x768 


How  many  total  combinations  are  there? 

What  is  the  minimum  number  of  these  combinations  we  will  have  to  test 
(and  which  ones  are  they)  in  order  to  determine  if  every  2-way  combination 
(e.g.,  Dell  Bios  with  Teac  CD)  will  indeed  work  properly  together? 

To  answer  this  question,  we  used  Pro-Test  software.  The  answer  is  25 
runs  and  those  25  combinations  are  shown  on  the  next  page. 


©  Philip  Mayfield,  Digital  Computations 


12 


Copyright 

2006 


Air 

Academy 

Associates 

Copyright 

2006 


High  Throughput  Testing  (HTT) 


(for  all  two-way  combinations) 


5  Levels 

Full  Factorial  = 

3  Levels  3  Levels 

8100  runs 

3  Levels 

HTT 

5  Levels 

=  25  runs 

3  Levels 

2  Levels 

2  Levels 

Motherboard 

RAM 

BIOS 

CD 

Monitor 

Printer 

Voltage 

Resolution 

Case  1 

ASUS 

256 

MB 

Dell 

Generic 

ViewSonic 

Lexmark 

110  V 

800  x  600 

Case  2 

Compaq 

512 

MB 

Dell 

Teac 

Sony 

HP 

220  V 

1024x768 

Case  3 

Gateway 

128 

MB 

Generic 

Sony 

KDS 

Cannon 

220  V 

800  x  600 

Case  4 

Dell 

128 

MB 

Award 

Teac 

NEC 

Cannon 

110  V 

1024x768 

Case  5 

Micronics 

256 

MB 

Generic 

Teac 

Generic 

Lexmark 

220  V 

1024x768 

Case  6 

Gateway 

256 

MB 

Award 

Sony 

Sony 

HP 

110  V 

1024x768 

Case  7 

Micronics 

512 

MB 

Award 

Generic 

ViewSonic 

Cannon 

220  V 

1024x768 

Case  8 

ASUS 

512 

MB 

Generic 

Teac 

KDS 

HP 

220  V 

1024x768 

Case  9 

Compaq 

128 

MB 

Award 

Generic 

Generic 

HP 

110  V 

800  x  600 

Case  1 0 

Micronics 

512 

MB 

Generic 

Teac 

Sony 

Lexmark 

110  V 

800  x  600 

Case  1 1 

Dell 

256 

MB 

Award 

Generic 

KDS 

Lexmark 

110  V 

1024x768 

Case  12 

Gateway 

512 

MB 

Dell 

Sony 

Generic 

Lexmark 

110  V 

1024x768 

Case  1 3 

Compaq 

256 

MB 

Generic 

Sony 

ViewSonic 

Cannon 

220  V 

1024x768 

Case  14 

ASUS 

128 

MB 

Dell 

Sony 

NEC 

Cannon 

220  V 

800  x  600 

Case  1 5 

Micronics 

128 

MB 

Dell 

Sony 

KDS 

Lexmark 

220  V 

800  x  600 

Case  1 6 

Gateway 

128 

MB 

Generic 

Teac 

ViewSonic 

HP 

110  V 

800  x  600 

Case  1 7 

Dell 

128 

MB 

Dell 

Sony 

Sony 

Cannon 

110  V 

1024x768 

Case  1 8 

ASUS 

256 

MB 

Award 

Sony 

Generic 

Cannon 

220  V 

1024x768 

Case  1 9 

Compaq 

512 

MB 

Dell 

Sony 

NEC 

Lexmark 

110  V 

800  x  600 

Case  20 

Gateway 

256 

MB 

Generic 

Generic 

NEC 

Cannon 

220  V 

800  x  600 

Case  21 

Micronics 

512 

MB 

Generic 

Teac 

NEC 

HP 

220  V 

800  x  600 

Case  22 

ASUS 

256 

MB 

Generic 

Generic 

Sony 

HP 

110  V 

800  x  600 

Case  23 

Dell 

512 

MB 

Generic 

Sony 

ViewSonic 

HP 

220  V 

1024x768 

Case  24 

Compaq 

256 

MB 

Dell 

Generic 

KDS 

Cannon 

220  V 

1024x768 

Case  25 

Dell 

128 

MB 

Generic 

Sony 

Generic 

HP 

110  V 

800  x  600 

Philip  Mayfield,  Digital  Computations 
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Test  and  Evaluation  for  Screening  and  Modeling 


Parameters 
or  Factors 
that 

Influence 
the  CTC 


r 


< 


v 


y  =fi  (X1.X2.X3) 

s  =f2(Xi,x2,  x3) 
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Where  does  the  data  for  evaluation  come  from? 

-  Design  of  Experiments  (Multivariate  Testing) 

-  Historical  Data  Analysis 

-  Simulation 
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Design  of  Experiments  (DOE) 

► 

•  An  optimal  data  collection  methodology 

r 

•  “Interrogates”  the  process 

\ 

•  Used  to  identify  important  relationships 
between  input  and  output  factors 

\ 

\ 

•  Identifies  important  interactions  between 
process  variables 

\ 

•  Can  be  used  to  optimize  a  process 

Air 
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Associates 

•  Changes  “1  think”  to  “1  know” 
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What  Is  a  Designed  Experiment? 


Purposeful  changes  of  the  inputs  (factors)  in  order  to  observe 
corresponding  changes  in  the  output  (response). 
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Inputs 


Outputs 


Run 

X,  X2  X3  X4 

CM 

>- 

Y 

SY 

1 

2 

3 
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DOE:  Determining  How  Inputs  Affect  Outputs 


► 


Air 
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i) 


Factor  A  affects  the  average 


y 

Factor  B  affects  the  standard  deviation 


y 


Factor  C  affects  the  average  and  the 
standard  deviation 


Di  D2 

y 
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Today's  Methods  of  Experimentation: 
Orthogonal  or  Nearly  Orthogonal  Designs 


FULL  FACTORIALS  (for  small  numbers  of  factors) 
FRACTIONAL  FACTORIALS 
PLACKETT  -  BURMAN 
LATIN  SQUARES 
HADAMARD  MATRICES 
BOX  -  BEHNKEN  DESIGNS 
CENTRAL  COMPOSITE  DESIGNS 


Taguchi  Designs 


SIMPLE  DEFINITION  OF  TWO-LEVEL 
ORTHOGONAL  DESIGNS 


Run 


Actual  Settings 


1 

2 

3 

4 

5 

6 

7 

8 


Coded  Matrix 


Response 
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DOE  “Market  Research”  Example 


Purpose: 

1.  Determine  the  relative  importance  of  product  attributes 
in  the  consumer  choice  process 

2.  Determine  the  composition  of  the  most  preferred  product 

3.  Estimate  market  share  of  a  given  product 

4.  Segment  the  market  as  to  their  preferred  product  profile 


Air 

Academy 

Associates 
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DOE  “Market  Research”  Example  (cont.) 


► 


Air 

Academy 

Associates 
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Suppose  that,  in  the  auto  industry,  we  would  like  to  investigate  the  following  automobile 
attributes  (i.e.,  factors),  along  with  accompanying  levels  of  those  attributes: 


A:  Brand  of  Auto: 

-1  =  foreign 

+1  =  domestic 

B:  Auto  Color: 

-1  =  light 

0  =  bright 

+1  =  dark 

C:  Body  Style: 

-1  =  2-door 

0  =  4-door 

+1  =  sliding  door/hatchback 

D:  Drive  Mechanism: 

-1  =  rear  wheel 

0  =  front  wheel 

+1  =  4-wheel 

E:  Engine  Size: 

-1  =  4-cylinder 

0  =  6-cylinder 

+1  =  8-cylinder 

F:  Interior  Size: 

-1  <  2  people 

0  =  3-5  people 

+1  >  6  people 

G:  Gas  Mileage: 

-1  <  20  mpg 

0  =  20-30  mpg 

+1  >  30  mpg 

H:  Price: 

-1  <  $20K 

0  =  $20-$40K 

+1  >  $40 K 

In  addition,  suppose  the  respondents  chosen  to  provide  their  preferences  to  product 

profiles  are  taken  based  on  the  following  demographic: 

J:  Age: 

-1  <  25  years  old 

+1  >  35  years  old 

K:  Income: 

-1  <  $30K 

+1  >  $40 K 

L:  Education: 

-1  <  BS 

+1  > 

BS 
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DOE  “Market  Research”  Example  (cont.) 


Question:  Choose  the  best  design  for  evaluating  this  scenario 


Answer: 


L18  design  with  attributes  A  -  H  in  the  inner  array  and 
factors  J,  K,  and  L  in  the  outer  array,  resembling  an 
L18  robust  design,  as  shown  below: 


Air 

Academy 

Associates 


Run* 


1 

2 

3 

4 

5 

6 

7 

8 

9 

10 
11 
12 

13 

14 

15 

16 

17 

18 


A  B  C  D 


F  G  H 


o 

+ 

0 

+ 

0 

+ 

0 

+ 

0 

+ 

0 

+ 


0 

+ 

0 

+ 

0 

+ 


0 

0 

+ 


L 

K 

J 


+  -  + 
+  + 


+ 

+  + 

+  + 


y 2  ys  Y4  y5  ye  Y7  Ys 


Segmentation  of  the  population  or 
Respondent  Profiles 


18  different  product  profiles 


21 


Copyright 

2006 


Catapulting  Power  into  Modeling 


► 
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Statapulf  Catapult 
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Catapulting  Power  into  Modeling 
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The  Theoretical  Approach 


lo0  =  rFF(0)sin0cos(p-(MgrG  +  mgrB)sin0 

0 


tan<|)  = 


D-rp  sin0 
d  +  rF  cos  0  ’ 


1 


* 

6b 


lo0  =  rFl  F(0)sin0 coscpd0-(MgrG  +mgrB)(sin0-sin0o) 


1 


JF 


l0 0^  =  rF  F(0)sin0coscpd0-(MgrG  +mgrB)(sin01  -sin0o). 


X  =  vB  cos 


eb 

tt  ^  1 

- 01  t — rB  cos  0. 

2  1  r\  d  1 

J  2 


(■ 


\ 


TT  1 

y  =  rBsin01  +  vB  sin  --01  t--gt2 

v2  J  2 


f  n  ^ 


rB  sin0-| +(R  +  rB  cos0-|)tan  — 0-| 

V2  J 


g  (R  +  rBcos0-|)' 
cos2 


v2  j 


=  0. 


Air 

Academy 
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gio 


(R  +  rB  cos  0-|  Y 


4rB  2 
D  cos 


2  01 

J 


fn  ' 

rB  sin01 +(R  +  rB  cos0-|)tan  — 0-| 

v2  J 


0| 


=  rFj*  F(0)sin0  cos(|)d0-(MgrQ  +mgrB)(sin0-|  -sin0o). 
% 
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Statapult  Exercise 

(DOE  demonstration) 


► 


Air 

Academy 

Associates 

Copyright 

2006 


Actual 

Factors 


Coded  Factors  Response  Values 


Run 

A  B 

A  B  AB 

Yi  Y2 

Y 

S 

1 

2 

3 

4 

Avg  - 

Avg  + 

A 

SV 

Y  = 
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Value  Delivery:  Reducing  Time  to  Market 

for  New  Technologies 


INPUT 


OUTPUT 


Pitch  <) 

(0,15,30) 

Roll  <) 

(0,15,30) 

Modeling  Flight 

W1F  <) 

(-15,0,15) 

Characteristics 

Six  Aero- 

W2F  <) 

(-15,0,15) 

of  New  3-Wing 

Characteristics 

W3F  <) 

(-15,0,15) 

Aircraft  * 

Air 

Academy 
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*  Patented  by  Dr.  Bert  Silich 


•  Total  #  of  Combinations  =  35  =  243 

•  Central  Composite  Design:  n  =  30 
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Aircraft  Equations 


► 
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CL  =  .233  +  .008(P)2  +  .255(P)  +  .012(R)  -  .043(WD1)  -  .117(WD2)  +  .185(WD3)  + 

.010(P)(WD3)  -  .042(R)(WD1)  +  .035(R)(WD2)  +  .016(R)(WD3)  +  .010(P)(R)  - 
.003(WD1  )(WD2)  -  .006(WD1)(WD3) 

CD  =  .058  +  .016(P)2  +  .028(P)  -  .004(WD1)  -  .013(WD2)  +  .013(WD3)  +  .002(P)(R)  - 
.004(P)(WD1)  -  .009(P)(WD2)  +  .016(P)(WD3)  -  .004(R)(WD1)  +  .003(R)(WD2) 
+  .020(WD1)2  +  .017(WD2)2  +  .021  (WD3)2 


CY  =  -.006(P)  -  .006(R)  +  .169(WD1)  -  .121(WD2)  -  .063(WD3)  -  .004(P)(R)  + 

.008(P)(WD1)  -  .006(P)(WD2)  -  .008(P)(WD3)  -  .012(R)(WD1)  -  .029(R)(WD2)  + 
.048(R)(WD3)  -  .008(WD1)2 

CM  =  .023  -  .008(P)2  +  .004(P)  -  .007(R)  +  .024(WD1)  +  .066(WD2)  -  .099(WD3)  - 
.006(P)(R)  +  .002(P)(WD2)  -  .005(P)(WD3)  +  .023(R)(WD1)  -  .019(R)(WD2)  - 
.007(R)(WD3)  +  .007(WD1)2  -  .008(WD2)2  +  .002(WD1)(WD2)  + 
.002(WD1)(WD3) 

CYM=  .001(P)  +  .001(R)-.050(WD1)  +  .029(WD2)  +  .012(WD3)  +  .001(P)(R)  - 

.005(P)(WD1)  -  .004(P)(WD2)  -  .004(P)(WD3)  +  .003(R)(WD1)  +  .008(R)(WD2)  - 
.013(R)(WD3)  +  .004(WD1)2  +  .003(WD2)2  -  .005(WD3)2 


.003(P)  +  .035(WD1)  +  .048(WD2)  +  .051  (WD3)  -  .003(R)(WD3)  +  .003(P)(R)  - 
.005(P)(WD1)  +  .005(P)(WD2)  +  .006(P)(WD3)  +  .002(R)(WD1) 
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Multiple  Response  Optimization 
Simulation  Example 
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Cross  Support 
Diameter  (2-4) 


Vertical  Support 
Diameter  (2-4) 


Box  Diameter  (2-4) 


Beam  Angle  (35-60) 


Vehicle 

Impact 

Safety 

Process 


Design  Cost  (<  300) 


Rib  Compression  (<  11) 


Triaxial  Acceleration  (<  15)^ 


©  Philip  Mayfield,  Digital  Computations 
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Iterative  Use  of  Simulation  and  Modeling 


► 
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©  Dr.  Srinivas  Kodiyalam,  SGI 
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Applying  Modeling  and  Simulation 
to  Automotive  Vehicle  Design 


Air 

Academy 

Associates 


IDENTIFY 

Many,  Many  x’s 

SCREENING  DESIGN 

The  critical 

CTCs, 

CDPs 

(DOE  PRO) 

few  CDP’s 

a 

k 

r 

_ t 

Examples  of  CTCs: 

NASTRAN  RADIOSS  MADYMO 

Safety  CTCs 
with  constraints 
specified  by  . 

FMVSS  \ 

(Federal  Motor 
Vehicle  Safety 
Standards) 


V. 


RAD  IOSS 

DYNA 

MADYMO 


no  federal 
requirements  y 
on  these  CTCs 


y1  =  weight  of  vehicle 
y2  =  cost  of  vehicle 
f  y3  =  frontal  head  impact 
y 4  =  frontal  chest  impact 
y5  =  toe  board  intrusion 
y6  =  hip  deflection 
y7  =  rollover  impact 
y8  =  side  impact 

r  y9  =  internal  aerodynamics  (airflow) 
y10  =  external  aerodynamics  (airflow) 

=  noise 

y12  =  vibration  (e.g.,  steering  wheel) 
y13  =  harshness  (e.g.,  over  bumps,  shocks) 
y14  =  durability  (at  100K  miles) 


Integrated  processes  with  high  fidelity 
CAE  analyses  on  HPC  servers 


Examples  of  Critical  Design  Parameters  (CDPs  or  Xs): 


J 


CFD 


x1  =  roof  panel  material 
x2  =  roof  panel  thickness 
x3  =  door  pillar  dimensions  =^>  i  beam 
x4  =  shape/geometry 
x5  =  windshield  glass 
x6  =  hood  material,  sizing  and  thickness 
x7  =  under  hood  panel  material,  sizing  and  thickness 


>  NASTRAN 
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Why  Robust  Design? 


X 


*• 


One  Input 


y 


> 


Air 

Academy 

Associates 


If  \ix  varies,  should  we  select  |n1  or  p2  to  hit  y  =  T? 


©  Philip  Mayfield,  Digital  Computations 
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Example  of  Iterative  Approach  to  Modeling  and 
Simulation  to  Optimize  Transmission  Performance 


Blue 


High  Fidelity 
Simulation  Model 

Low  Fidelity 
Analytical  Model 


Quadratic 

Response 


Surface  on  29 
Variables 


Monte  Carlo  Simulation 
with  various  weightings 
of  responses 


LHS:  Latin  Hypercube  Sampling  -  a  method  for 
sampling  a  design  space  when  deterministic 
simulation  is  being  used  in  high  dimensions, 
involving  many  input  variables. 


25-Level  LHS 
inserted  Noise  in 
each  of  8  Noise 
Variables  and  21 
Design  Variables 
(testing  the 
designs  in  the 
presence  of  noise) 


Also,  reduced  to 
11  Design 
Variables  plus  8 
Noise  Variables 


Air 
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LHS  used  to  generate  5-rep  outer 
array  from  8  Noise  Variables 


Phase  2 


1 1  Design  Var. 

8  Noise  Var. 


Quadratic  Model , 


y  model 
s  model 


DOE  Pro 
Multiple 
Response 
Optimization 


CCD  Inner  Array 
Alters  1 1  Design 
Variables 


25-Level  LHS  inserted 
Noise  in  each  of  8 
Noise  Variables  and  21 
Design  Variables 
(testing  the  designs  in 
the  presence  of  noise) 


Build  and 
Test 

Prototype 


Philip  Mayfield,  Digital  Computations 


Modeling  the  Simulator 


► 


\ 
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Critical  Parameters  ID'd 


Transfer  Function  on 
v  Critical  Parameters 

Perform  DFSS  Activities  (Robust 
Design,  Tolerance  Allocation) 

Using  Transfer  Function 

Optimized  Design 


>r 


Optimized  Simulator 

_ y  _ 

Build  Prototype  to  Validate 
Design  in  Real  World 

36 
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Environments  Where  Simulation  and 
Modeling  Is  Beneficial 


A  high  number  of  design  variables 
A  substantial  number  of  design  subsystems  and 
engineering  disciplines 

Interdependency  and  interaction  between  the  subsystems 
and  variables 

Multiple  response  variables 

Need  to  characterize  the  system  at  a  higher  level  of 
abstraction 

Time  and/or  space  must  be  compressed 
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The  Vision  of  DFSS 


► 
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Reactive 
Design  Quality 


DFSS 


Predictive 
Design  Quality 


From 

To 

•  Evolving  design 

•  Disciplined  CTC 

requirements 

flowdown 

•  Extensive  design 

•  Controlled  design 

rework 

_  l\ 

parameters 

•  Product  performance 

•  Product  performance 

assessed  by  “build 

V 

modeled  and  simulated 

and  test” 

•  Performance  and 

•  Designed  for  robust 

producibility 

performance  and 

problems  fixed  after 

producibility 

product  in  use 

•  Quality  “tested  in” 

•  Quality  “designed  in” 
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For  Further  Information,  Please  Contact: 
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Air  Academy  Associates,  LLC 

1650  Telstar  Drive,  Ste  110 
Colorado  Springs,  CO  80920 


Toll  Free:  (800)  748-1277  or  (719)  531-0777 
Facsimile:  (719)  531-0778 
Email:  aapa@airacad.com 
Website:  www.airacad.com 


Air~ 

Academy 

Associates 
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Headquarters  U.  S.  Air  Force 


Integrity  -  Service  -  Excellence 


The  Air  Force’s  Perspective 


Mr.  Jack  Manclark 

Phone:  703-697-4774  or  DSN  227-4774 
E-mail:  John.Manclark@pentagon.af.mil 
NDIA,  Jacksonville,  FL,  8  Mar  06 

U.S.  AIR  FORCE 


Modeling  and  Simulation  (M&S) 


U.S.  AIR  FORCE 


■  M&S  -  “The  Good ” 

■  F/A-22 

■  Ranges 

■  Hardware  in  the  Loop 

■  M&S  -  “The  Bad’ 

■  Complex  Threat  Simulators 


3/13/2006 


Integrity  -  Service  -  Excellence 
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M&S-A  Word  of  Caution 


U.S.  AIR  FORCE 


■Joint  Modeling  and 
Simulation  System  (JMASS) 

■  Requirements  Creep 

■  We  built  but  they  did  not  come 

■  Blue  Model  Challenge 


3/13/2006 


Integrity  -  Service  -  Excellence 
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T&E  Resources 


U.S.  AIR  FORCE 


■  National  Defense  Authorization 
Act  (NDAA)  ’03 

■  Support  Contractor  Reduction 
(25%)  over  FYDP 

■  Personnel  Reductions 

■  Smart  Operations  21  (SO-21) 


3/13/2006 


Integrity  -  Service  -  Excellence 


4 


F-35  JSF  Testing  Challenges 


U.S.  AIR  FORCE 


■  International  Partner  Participation 

■  Desired  Level  of  Involvement 

■  Data  Requirements 

■  Security  Guidelines 


3/13/2006 


Integrity  -  Service  -  Excellence 
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U.S.  AIR  FORCE 


Initial  Operational  test  and  Evaluation 

(IOT&E) 


■  “Effective”  -  adequate  to  accomplish  a 
purpose 

■  “Not  Effective”  - 

■  Experience 

■ JSTARS 
■  Predator 


3/13/2006 


Integrity  -  Service  -  Excellence 
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Defense  Acquisition  Performance 
Assessment  (DAPA)  Report ,  Jan  06 


U.S.  AIR  FORCE 


■  Performance  Improvement 

1.  Make  operational  testing  more  realistic,  time  and  resource 
constrained,  and  limited  in  its  ability  to  create  additional 
performance  requirements. 

2.  Create  a  new  category  for  Initial  Operational  Test  and  Evaluation 

results  that  allows  Combatant  Commanders  to  accept  useful 
capabilities  for  deployment  which  the  Director  of  Operational  Test  and 
Evaluation  would  otherwise  determine  to  be  Not  Operationally 
Effective. 

3.  Require  that  test  planning  and  criteria  development  for  Operational 
Test  and  Evaluation  reflect  testing  in  environments  and  against  the 
range  of  threats  that  are  identified  by  the  Combatant  Commander 
--  not  by  the  test  community. 

4.  Give  Program  Managers  explicit  authority  to  defer  non-Key 
Performance  Parameter  related  requirements  to  later  acquisition  .  .  . 

5.  Reguire  Joint  Reguirement  Oversight  Council  approval  of  all  test 
plans  that  require  operational  testing  in  environments  other  than  those 
established  in  the  Test  and  Evaluation  Master  Plan. 


3/13/2006 


Integrity  -  Service  -  Excellence 
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U.S.  AIR  FORCE 


Critical  Operational  Issues  (COI) 
Rating  Definitions 


COI  Rating 

Definition 

Fully  Mission 
Capable 

(FMC) 

Mission  can  be  accomplished  in  the  intended 
operational  environment 

Mission  Capable 
(MC)  G 

Mission  can  be  accomplished,  but  with  increased 
operational  cost 

Partially  Mission 
Capable 

(PMC)  Y 

Only  some  aspects  of  mission  can  be  accomplished 

and/or 

Mission  can  only  be  accomplished  under  some 
conditions 

Not  Mission 
Capable 

(NMC) 

Mission  cannot  be  satisfactorily  accomplished 

3/13/2006 


Integrity  -  Service  -  Excellence 
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COMOPTEVFOR 

Perspectives 

on 

Modeling  &  Simulation 

in 

Operational  Test  &  Evaluation 


RDML  Bill  McCarthy 
8  March  06 


1 


Topics 


•  Current  Trends  in  Navy  Operational  Test 

•Integration  of  Testing  (DT/OT) 
•Enterprise  Solutions 

•  Need  for  Modeling  &  Simulation 

•  Policy  Considerations 

•  Challenges  -  Myths  and  Money 

•  The  Way  Ahead  -  again! 
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Current  Trends 


•  Integration  of  Test  and  Evaluation 

-  Fundamental  concept  is  to  minimize  the  duplication  of 
effort  by  identifying  common  data  requirements  up  front. 

•  Re-structured  MV-22  Program  is  an  example 

-  Ongoing  effort,  formalized  in  mid-2005  with  the 
development  of  an  Integrated  Test  Framework 

•  Common  test,  shared  data,  independent  analysis 

•  Reduce  cycle  time  and  cost  for  testing  while  providing  earlier 
operational  input 

-  Independent  OPEVAL  is  retained  to  ensure  statutory 
independence  of  the  Operational  Test  Agency;  however, 
the  scope  of  the  OPEVAL  can  be  reduced  to  the  extent  that 
valid  data  are  collected  from  integrated  test. 
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Current  Trends 


•  Enterprise  Approaches  to  Test  &  Evaluation 

-  Should  really  be  titled  Enterprise  Approaches  to  Systems  Engineering 

•  Logical  consequence  of  Family  of  Systems  development 

-  Ship’s  Self-Defense  System  is  proto-typical  example 

•  CVN-76/LPD-17/LHA-6/DD(X)/CVN-78 

-  Metric  for  AAW  assessment  is  Probability  of  Raid  Annihilation 

-  LHA-6  test  program  provided  the  forcing  function 

-  Individual  testing  of  the  full  combat  system  by  each  platform  would 
have  been  prohibitively  expensive 

-  No  single  program  could  bear  the  cost  of  a  Self-Defense  Test  Ship 
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Current  Trends 


•  Alternative  approach  -  realigned  development  and 
testing  program  under  PEO  IWS 

-  By  combining  test  objectives  across  platforms, 
conservative  estimate  is  a  $200M  reduction  in  missile  and 
target  costs 

-  Self-Defense  Test  Ship  will  allow  the  acquisition  of  data  to 
populate  the  models  needed  to  assess  PRA. 

•  Additional  benefits  anticipated  from  increased 
information  sharing  across  participating  platforms 
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An  Enterprise  Approach  to 
AAW  Self-Defense 


Lead 

Ship 


FY  06/07 


RAM 


SPQ9B 

SPQ  9B 

SPS  48 

SPS  49 

SSDS  MK  2 

MK9TI 

CEC 

SSDS  MK  2 

FY  08/09 


RAM 

ESSM 


RAM 


SDTS 


|: 

2  | 


SPQ9B 

RADAR 

SPS  49* 

C2  SYSTEM 

MK  9TI* 

SSDS  MK  2 

FY  10/11 


SPQ  9B 
SPS  49 
SPS  48E? 
SEWIP 


FY  11/12 


RAM  P3I/ 
ESSM 


FY  12/13 


ESSM 


SPQ9B 
SPS  49 
SPS  48E 
SEWIP 
CEC 

SSDS  OACE 


DBR 

TSCE 


FY  14/15 


RAM  P3I/ 
ESSM 


MFR 

TSCE** 


|  Lead  Ship 

1  B  SDTS 

s 

am 

isted 

|  |  First  Test  of  Thi 

Integrated  Syst< 

1  1  Derivative  of  Te 

Articles 

*  CVN  Only 
?  Test  Results  apply  tc 
other  SSDS  Variants 
**  Core  TSCE  function; 
with  additional 
applications 

) 

ality 

Need  for  Modeling  &  Simulation  in 
Operational  Test  and  Evaluation 


•  21st  Century  warfare  systems  are  required  to 
operate  in  complex  environments  that  are 
difficult  to  assess 

-  AAW  performance  assessment 

•  Need  to  assess  multiple  hard  and  soft-kill  systems 
working  together 

-  Electronic  Warfare  systems 

•  Realistic  pulse  densities;  unique  threat  emitters 

-  Undersea  Warfare  systems 

•  Multiple  environmental  conditions;  realistic  targets 
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Need  for  Modeling  &  Simulation  in 
Operational  Test  and  Evaluation 


•  Put  another  way,  anything  short  of  actual  use  in 
combat  is  to  a  greater  or  lesser  extent  a  form  of 
modeling  or  simulation. 

•  We  have  neither  the  time  nor  the  money  to  build  large 
numbers  of  threat  replicators  necessary  to  test  the 
performance  of  a  systems  of  systems  in  the  diverse 
environmental  conditions  that  may  be  encountered. 

•  The  challenge  is  to  find  the  right  mix  of  M&S  and 
live  end-to-end  testing  to  ensure  that  weapon  systems 
will  perform  as  predicted  in  actual  combat. 
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Policy  Considerations 


•  DODD  5000. 1  -  “The  conduct  of  test  and  evaluation, 
integrated  with  modeling  and  simulation,  shall  facilitate 
learning,  assess  technology  maturity  and  interoperability, 
facilitate  integration  into  fielded  forces,  and  confirm 
performance  against  documented  capability  needs  and 
adversary  capabilities. . .”  (El .  1 1) 

•  DODI  5000.2- 

-  “The  T&E  strategy  shall  provide  information  about  risk  and  risk 
mitigation,  provide  empirical  data  to  validate  models  and  simulations, 
evaluate  technical  performance  and  system  maturity,  and  determine 
whether  systems  are  operationally  effective,  suitable,  and  survivable 
against  the  threat  in  the  System  Threat  Assessment.”  (E5.1.1) 

-  “Appropriate  use  of  accredited  models  and  simulation  shall  support 
DT&E,  IOT&E,  and  LFT&E.”  (E5. 1.4.7) 
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Policy  Considerations 


•  SECNAVINST  5000. 2C  -  .  .M&S  may  be  used  during  T&E 

of  an  ACAT  program  to  represent  conceptual  systems  that  do 
not  exist  and  existing  systems  that  cannot  be  subjected  to 
actual  environments  because  of  safety  requirements  or  the 
limitations  of  resources  and  facilities.  M&S  applications 
include  hardware/software/operator-in-the-loop  simulators, 
land-based  test  facilities,  threat  system  simulators,  C4I  systems 
integration  environments/facilities,  and  other  simulations  as 
needed.  M&S  shall  not  replace  the  need  for  OT&E  and  will 
not  be  the  primary  evaluation  methodology.  M&S  shall  not  be 
the  only  method  of  meeting  independent  OT&E  for  beyond 
low  rate  initial  production  (BLRIP)  decisions  per  USC  2399. 
M&S  is  a  valid  T&E  tool...”  (5.4.7.9) 
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Challenges  -  Myths  and  Money 


•  Perceptions 

-  M&S  is  an  inexpensive  substitute  for  testing. 

-  M&S  is  the  natural  extension  of  the  computer  gaming 
phenomenon. 

-  M&S  will  revolutionize  acquisition. 

•  Facts 

-  M&S  can  provide  information  about  system  performance 
under  a  variety  of  conditions  that  can  not  be  practically 
assessed  with  live  testing. 

-  Development  of  models  is  a  complex  engineering  task. 
Models  and  simulations  vary  greatly  based  upon  their 
purpose. 

-  M&S  is  an  essential  component  in  evolutionary  acquisition? 


The  Way  Ahead  -  Again! 


•  There  are  few,  if  any,  new  ideas  needed  to  make 
M&S  a  more  effective  tool. 

-  In  the  last  8  years  there  have  been  a  variety  of  studies,  the 
need  is  not  for  study  but  implementation. 

•  M&S  has  played  a  critical  role  in  the  development 
and  operational  testing  of  EW  systems  for  decades. 

-  We  need  to  learn  from  this  experience  and  use  the  right 
type  of  M&S  where  it  best  fits. 

-  Successful  use  requires  a  rigorous  understanding  what  the 
particular  form  of  M&S  can  bring  to  the  program. 
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The  Way  Ahead  -  Again! 


M&S  must  be  addressed  in  the  T&E  Strategy  and  the  TEMP. 

-  The  integrated  test  team  needs  to  determine  where  various  M&S  tools 
are  best  suited  for  use. 

-  M&S  needs  to  be  understood  as  a  tool  set,  with  a  variety  of  different 
tools,  each  suited  for  different  applications. 

Program  managers  must  make  timely  investments  to  develop 
the  models  and  collect  the  data  necessary  for  viable  M&S 
tools. 

-  Even  when  modeling  is  used,  too  many  programs  reach  the  completion 
of  DT&E  without  completing  the  verification  and  validation  of  the 
models  used. 

Enterprise  solutions  require  Enterprise  level  investments  in 
appropriate  tools,  such  as  the  Self-Defense  Test  Ship. 

-  Without  an  Enterprise  approach,  M&S  tools  are  not  likely  to  be 
available  in  time  to  support  key  acquisition  decisions  for  “systems  of 
systems”. 
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The  Way  Ahead  -  Again! 


•  Current  policies  clearly  support  the  use  of 
M&S  throughout  the  entire  test  and  evaluation. 

•  Rather  than  new  policy,  we  need  to  enforce  a 
disciplined  systems  engineering  approach  that 
holds  developers  accountable  for  using  all 
available  tools  to  best  understand  the 
capabilities  and  limitations  of  the  weapon 
system  being  developed  for  the  warfighter. 
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Questions? 


Challenges  For  M&S  in  Army 
Test  and  Evaluation 


Roe  Mirabelle 

Chief,  Technical  Support  Division  (TSD) 

Modeling  &  Simulation 
410-306-0403 


Modeling  and  Simulation  in  Support  of  Army  Transformation 

Quality  Control  for  the  US  Army 


Army  Test  and  Evaluation  Command 


Challenges  For  M&S  in  T&E  :  Outline 


>  Reality  of  T&E  Transformation 


>  Getting  Your  Needs  Identified:  Plan  &  Execute 

>  Requirements  Identification  and  Decomposition 

>  T&E 

>  M&S 

>  Architecture 

>  V&V  -and  “A” 


>  Lack  Of  Success— From  Some  Experts! 

>  Challenge  to  T&E--Make  it  Better? 


Army  Test  and  Evaluation  Command 


ATEC’s  T&E  Mission  is  Changing 

Transformation  of  Test  and  Evaluation 


Yesterday 

Test  &  Evaluation 


System  Design  Moving  Toward 
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•Greater  S/W  Based  Design 

•Technological  Improvement 

•More  Complex  FoS  and  SoS  Interactions 


Test/simuiation  &  Evaluation 


Test/Simulation  &  Evaluation 


Simulation/Test  &  Evaluation 


Tomorrow 


Bottom  Line: 

We’ll  always  test  hardware,  but  are  seeing  an  increased  reliance  on  M&S 
which  will  have  a  more  prominent  role  in  our  evaluations  and 

increase  our  need  for  technical  expertise. _ 3 


Army  Test  and  Evaluation  Command 


Challenge  Facing  ATEC 


Providing  fully  integrated  T est  &  Simulation  Evaluations. 


To  accomplish  that  mission  ATEC  must  develop  and  maintain  adequate  technical  expertise 
to  support  the  command  in  the  following  areas: 


What  we  evaluate 


•  Technology  Development  -  Integrating  FoS  and  SoS 
technologies  and  architectures  to  develop  comprehensive 
evaluation  approaches. 


How  we  evaluate 


•  Modeling  and  Simulation  -  Tracking  the  development, 
VV&A  of  models. 

•  Employing  Model-Test-Model  to  evaluation  phases  (pretest- 
— test — post  test) 

•  Experimental  Design  -  Designing  tests  that  make  the  most 
out  of  expensive  hardware  testing  while  taking  into 
consideration  how  M&S  will  be  used. 
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Requirements  Planning  Process 


The  KEY  to  being  successful  is  approved  requirements  —  driving  the  evaluation  focus  areas. 
If  requirements  are  changing — a  ripple  effect  occurs  at  every  level  of  the  process. 


Army  Test  and  Evaluation  Command 


Test  &  Evaluation  Planning  Process 
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Army  Test  and  Evaluation  Command 


View  of  M&S  Development  Process  For  FCS 


SOS  M&S  Reqs 


SVF  Reqs 
Panel 


Mgmt 


Exec 


SME 


Auth 


Approved  / 
Prioritized 
Reqs. 


SOS  M&S  Development 

M&S  IPT  SOS  M&S  VV&A 
Panel 


S2F/FSE  Application 


SVF  Build  l...n 


Mgmt 


Exec 


SME 


Auth 


SOS  Level 


ww 


Applications: 

Spec  Demo 
|  |  ORD 

[Jess 


System  Level 

■ 


LSI  System 
Engineering 
Verification  & 
Test  Process 


ORD 

O&O 


Integrated 

Verification 

Method(s) 


Test 


|  Plan  Together,  Test  Once, 
Distribute  Data 


Analysis 


_,_0_  r-«  Integrated 

M&S  Reqs  Requirements 


Government 
Evaluation  & 

SEP 

► 
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Test  Process 
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Demo 
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Inspect 
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Stimulators, 

Emulators, 

Instrumentation 


M&S 

Components/F  unctions 
for  Service-based 
Architecture 


Panel  Process 


Simulators, 
Analysis  M&S 


Participants: 

|  |  PM  /  LSI 

■  ATEC 
■ UAMBL 
I  I RDECOM 
i  I  DEVELOPERS 


v&v  By  Provider 


Candidates 
M&S  from 
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System 
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VI  1  Q/1Q/m 


Army  Test  and  Evaluation  Command 


TEMP,  SIP 
LFDs,  IPs 


Getting  M&S  Requirements  Allocated 


Slice  through 


Functional  Decomposition 
Capabilities  Analysis 


S1  ~-:-=  -r* 

<id  *  ■■  - 


Yields 


Phases  Steps  per 
Engineering  Iteration 


ft" 


(*P*-  aSS 


Used  to 
Annotate 


Requirements  per  phase 


Iteration  Specific  SV-5  views 


Generates 
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Army  Test  and  Evaluation  Command 


The  W&A  Process:  Lots  to  Do  Over  Time 


Program 
Level 


Mgmt. 

Level 


Acoreditatiorli 


Data 

Collection 

Events 


>1 

1  1 

r  V 

Validation 

w 

Model 

Events 


Model  Selection  &  Upgrades 


Input  Database 


Enhanced  Model 
&  Database 


valuation 


Evaluator 

Events 


*SOS  Evaluation 


'est  Planning 

Accred 

Support 

Package 

MOE  Evaluation. 

SOS  Inputs 

Army  Test  and  Evaluation  Command 


High  Level  M&S  Challenges  (1  of  2) 


>  Incorporating  requirements  into  tracking  and  allocation  database. 

>lf  a  requirement  is  uniquely  government,  who  is  responsible  for  that  integration? 
>ls  the  allocation  decomposition  completed? 

>  Overlap  between  test  execution  and  development/execution  of  the  next 
phase. 

>  Is  there  sufficient  time  to  implement  required  fixes  or  apply  lessons  learned  in 
schedule? 

>  Bow  wave  effect  of  pushing  off  M&S  requirements  to  future  phases. 

>  How  early  are  M&S  (and  T&E)  requirements  needed  to  be  identified? 

>  2  years  before  execution? 

>  Ability  to  verify  and  validate  (and  increase  risk)  as  schedules  slip? 
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Army  Test  and  Evaluation  Command 


High  Level  M&S  Challenges  (2  of  2) 


>  Changes  in  plans  regarding  M&S  in  test  augmentation. 

> Example:  ability  to  test  tactical  SW  in  emulators  and  collect  V&V  artifacts 

>Battle  Command  Simulation  Needed?  Yes  or  NO? 

>  Representation  of  the  fully  integrated  Battle  Command  capability. 
Available  when?  Final  delivery  of  the  actual  code  (Phase  4/Post  MS  C)? 

>  Can  incremental  deliveries  of  tactical  software  integrated  into  the 
simulation  environment  be  used?  Available? 

>  Ability  to  TEST  all  of  the  systems  together  within  a  full  SoS  context. 

Test  articles  with  C4ISR  and  full  functionality  available  when? 

>  Ability  of  M&S  to  represent  full  FCS  BCT.  Available  when? 

>A  V&V  Plan — (versus  a  strategy)--  for  M&S  tools  is  required. 

>  Accreditation  criteria  needed  (ATEC  Action) 

>  V&V  Plan  needed  (PM-Action) 
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Army  Test  and  Evaluation  Command 


Example  of  M&S  Challenge  Affecting  Evaluation 


>  Demonstrate  low  risk  of  KPP  completion  at  Milestone  C 

>  FCS  BCT  consists  of  three  battalions,  cannot  realize  this  complexity  in  actual  hardware 

>  Virtual  and  constructive  simulation  capability  critical 

>  KPP’s  Not  Possible  without  the  network 

>  Must  objectively  load  the  network  for  results  to  be  credible 

>  Data,  voice,  video,  messaging,  etc.. 

>  M&S’s  critical  function  is  to  provide  the  “wrapper”  or  stimulus  for  the  objective 
simulation/software 

>  Must  provide  an  environment  from  where  system  and  SoS  capabilities  may  be 
demonstrated 

>  Terrain,  weather,  lethality  brokerage,  etc... 

>  Non-intrusive  network  simulation  and  credible  aggregate  force  representations 
are  of  the  highest  priority 

>  Critical  to  provide  “wrap”  for  the  equipment  on  the  range 


>  Non-intrusive  test  network  for  testing  at  system,  FCS  BCT,  Complementary  and 
Joint  systems 
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Army  Test  and  Evaluation  Command 


M&S— Not  Worked  as  Often  as  Liked 

>  2003  Academy  Industry  Symposium  on  Mathematical  Modeling  discussed 
reasons  for  the  lack  of  success  in  use  of  models: 

>  Inappropriate  method  of  formulating  or  representing  the  problem 

>  Ill-conceived  multi-disciplinary  integration 

>  Inability  to  communicate  among  multi-disciplinary  team  members 

>  Few  existing  value/reward  systems  actually  reward  collaboration 

>  Funding  sources  not  deal  well  with  multi-disciplinary  work 

>  Isolation  of  modeling  community 

>  Difficulty  in  integrating  models  of  different  resolution 

>  The  practical  sociology  between  modelers  and  decision  makers: 

>  Modeling  should  help/be  an  aid  to  critical  thinking 


EXCERPT  FROM:  PHALANX  Article,  Sep  2003: 
“Issues  in  Model  Integration  to  Support  Decision 
Makers”;  by  Dr.  Ernest  Seglie 
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Army  Test  and  Evaluation  Command 


M&S 

Not  Worked  as  Often  as  Liked 

2003  Academy  Industry  Symposium  on  Mathematical  Modeling  discussed  reasons  for  the  lack 
success  in  use  of  models: 


>  Inappropriate  method  of  formulating  or  representing  the  problem 

>  Requirements  Identification  process  cumbersome  (Subcontractors,  Prime,  Engineering, 
Test,  Evaluation) 

>  T&E  issue  is-can  we  live  with  less  testing?  So,  we  become  a  user  of  these  products 

>  Example:  JTRS:  supporting  development  of  FCS  NW  for  Current  Force  and  Future  Force 

>  Modeling  while  development  ongoing.  Objectives  are? 

>  Is  transport  layer  for  FCS;  yet  architecture  definition  evolving. 

>  Planned  integrated  approach  through  PM  M&S  team,  but  harder  to  say  what  we  can 
bank  on 


>  Ill-conceived  multi-disciplinary  integration 

>  So  many  systems  +  Network;  Modeling  and  development  ongoing  simultaneously 

>  Have  SOS  to  address;  Who  has  done  this  before— at  a  level  of  required  fidelity? 

>  Inability  to  communicate  among  multi-disciplinary  team  members 

>  Players:  Subs  +  LSI  +  ATEC  +  Evaluation  (SEP)  +  PM  +  CTO  +  PM  M&S  (MSMO)  + 
Training 

>  T&E  challenge  to  program  is:  Communicating  requirements  for  this  SOS  M&S  to  this 
hugh  crowd  -  to  include  JOINT— with  varying  crowd  experience-NOT  EASY! 

>  What  does  program  integrator  know  about  how  ARMY  does  SOS  testing?  Evaluation? 


EXCERPT  FROM:  PHALANX,  Sep  2003:  “Issues  in  Model  Integration  to  Support  Decision  Makers”;  by  Dr.  Ernest  Seglie 
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M&S 

Not  Worked  as  Often  as  Liked  (continued) 

2003  Academy  Industry  Symposium  on  Mathematical  Modeling  discussed  reasons  for  the  lack 
success  in  use  of  models: 


>  Few  existing  value/reward  systems  actually  reward  collaboration 

>  Stove  pipe  development;  scramble  for  dollars  to  develop  products;  parochial  views 

>  Funding  sources  not  deal  well  with  multi-disciplinary  work 

>  Program  dollar  cuts;  is  sudden  at  times. 

>  Other  priorities  evolve  and  take  over 

>  SW  development— costly  and  difficult 

>  Isolation  of  modeling  community 

>  What  IS  out  there  that  can  provide  best  insights  into  effectiveness,  suitability, 
survivability? 

>  Difficulty  in  integrating  models  of  different  resolution 

>  Major  integration  effort  on  this  program;  multiple  BOS  systems 

>  The  practical  sociology  between  modelers  and  decision  makers: 

>  Modeling  should  help/be  an  aid  to  critical  thinking... 


EXCERPT  FROM:  PHALANX,  Sep  2003:  “Issues  in  Model  Integration  to  Support  Decision  Makers”;  by  Dr.  Ernest  Seglie 
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Challenge  to  T&E--Can  We  Make  it  Better? 

Summary 

ncourage  programs  to  integrate  modeling  into  their  development 
and  decision  process. 


>  Integrate  into  T&E  evaluation  ONLY  IF  modeling  has  demonstrated 
it  is  an  integral  part  of  the  program  processes  and  demonstrates  it 
is  an  aid  to  critical  thinking. 

>  Multi-disciplinary  team  established,  in  place,  and  functioning  to  add 
information  to  contractors,  PM,  and  evaluators.  Needed  7-10  years 
before  IOT. 


>  Feedback  loop  working  through  cycle  on  early  component  testing 
and  early  operational  assessments. 


>  M&S  developed  in  sufficient  resolution  so  that  test  planning 
parameters  can  be,  and  are,  calculated  using  the  model. _ 

EXCERPT  FROM:  PHALANX  Article,  Sep  2003: 
“Issues  in  Model  Integration  to  Support  Decision 
Makers”;  by  Dr.  Ernest  Seglie 
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Army  Test  and  Evaluation  Command 


Conclusion: 

Simply  the  place  where  someone  got  tired  of  thinking. 
Great  Quotes  From  Great  Skeptics 


•  Well  informed  people  know  it  is  impossible  to  transmit  the  voice  over  wires  and  that  were  it 
possible  to  do  so,  the  thing  would  be  of  no  practical  value.  Editorial  in  the  Boston  Post  (1865) 


•  That  the  automobile  has  practically  reached  the  limit  of  its  development  is  suggested  by  the  fact 
that  during  the  past  year  no  improvements  of  a  radical  nature  have  been  introduced. 

Scientific  American,  Jan.  2, 1909 


•  Heavier-than-air  flying  machines  are  impossible. 

Lord  Kelvin,  ca.  1895,  British  mathematician  and  physicist 

•  While  theoretically  and  technically  television  may  be  feasible,  commercially  and  financially  I 
consider  it  an  impossibility,  a  development  of  which  we  need  waste  little  time  dreaming. 

-  Lee  DeForest,  1926  (American  radio  pioneer) 

•  There  is  not  the  slightest  indication  that  [nuclear  energy]  will  ever  be  obtainable.  It  would  mean 
that  the  atom  would  have  to  be  shattered  at  will.  Albert  Einstein,  1932. 


•  Where  a  calculator  on  the  ENIAC  is  equipped  with  19,000  vacuum  tubes  and  weighs  30  tons, 
computers  in  the  future  may  have  only  1,000  vacuum  tubes  and  perhaps  only  weigh  1.5  tons. 
Popular  Mechanics,  March  1949.  (Try  the  laptop  version!) 
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26  27  Steps  to  Success: 

The  DoD  Acquisition  M&S  Master  Plan 


22nd  Annual  National  Test  &  Evaluation  Conference 

Fred  Myers,  OUSD  (AT&L)  DS/SE 
Chair,  Acquisition  M&S  Working  Group 


March  9,  2006 


M&S  is  a  Necessary  Part  of  Acquisition 


M&S  is  broadly  useful  to  enable  systems  engineering 
throughout  a  system  or  S-o-S  life  cycle 


User  Requirements 
&  Concept  of 
Operations 


System 

Requirements  & 
Architecture 


Recursive 

Processes 


System 

Demonstration  & 
Validation 


System  Integration 
&  Test 


Component  Design  ]nt^Tkst 


~zr 


Procure,  Fabricate, 
&  Assemble  Parts 


Recursive 

Processes 


"Help  drive  good  systems  engineering  practices 
back  into  the  way  we  do  business.” 

Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics 


Acquisition  M&S  Working  Group 

Established  by  SE  Forum  Feb  2005,  to  define  a  plan  to  solve  the  problems 

...anchored  in  acquisition  community, 
linked  to  industry  and  M&S 


Industry  /  Academia 


DoD  SE 


DoD  M&S 


Chair:  Fred  Myers 


Products 

Reports  (e.g.,  “M&S  Support 
to  the  New  DoD  Acq.  Process”), 
standards,  papers,  etc. 


Product 

Acquisition  M&S 
Master  Plan 


Product 

DoDI  to  Implement 
DoDD  51 34.01 


A  Sampling  of  DoD’s  M&S  Problems 


□  Minimal  reuse  of  M&S  &  data;  hard  to  discover  reusable  resources; 
insufficient  info  to  evaluate  reuse  candidates;  no  incentives  to  re-use 

□  No  contractual  guidelines  for  M&S  and  data  needs 

□  No  DoD  requirement  for  formal  M&S  planning  to  support  acquisition 


□ 


Lack  of  agreed  standards  for  interchanging  information 


□  No  conceptual  framework  for  data  interchange 

□  Many  M&S  tool  gaps  and  deficiencies 

□  M&S  developers  -  not  M&S  users  -  tend  to  drive  M&S  development 

□  No  readily-available  L-V-C  distributed  environment 

□  VV&A  often  weak  or  non-existent;  documentation  inconsistent 

□  No  business  model  for  developing,  using,  maintaining  M&S  capabilities 

□  Acquisition  community  managers  (and  staffs)  are  mostly  uninformed  about 
M&S  capabilities  and  limitations 

□  Body  of  knowledge  for  M&S  support  to  acquisition  is  deficient,  not 
managed 


Acquisition  M&S  Master  Plan 
Development  Process 


□  Bottoms  Up: 

>  Assessed  recommendations  from  prior  studies  on 
M&S  in  Acquisition 

>  Assessed  current  issues  and  needs  (today’s  SoS  efforts) 

□  Top  Down  (Desired  Acquisition  Environment): 

>  DoDD  5000.1 

>  CJCSI  3170.01 

□  Identified  gaps  to  better  enable  Acquisition  M&S 

□  Identified  actions  to  address  the  gaps  for  Acquisition  M&S 

□  Identified  actions  of  others  (e.g.,  DMSO,  Nil,  Services) 

□  Documented  actions,  lead  organizations,  and  completion 
dates  in  the  draft  Acquisition  M&S  Master  Plan 


27  Steps  to  Success 


□  27  Needed  Actions 

□  Each  Action 

>  Rationale  (Why  is  it  needed?) 

>  Discussion  (What  is  the  specific  action?) 

>  Lead  &  Supporting  Organizations  (Who  is  involved?) 

>  Products  (What  is  the  product?) 

>  Completion  Goal  [year]  (When  will  the  action  be  completed?) 

□  Execution  Management 
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5  Acquisition  M&S  Objectives;  27  Actions 


Objective  1 


Provide 
necessary 
policy  and 
guidance 


Actions 

1.  M&S 
management 

2.  Model-based 
systems 
engineering  & 
collaborative 
environments 

3.  M&S  in  testing 

4.  M&S  planning 
documentation 

5.  RFP  &  contract 
language 

6.  Security 
certification 


Objective  2 


Enhance  the 
technical 
framework 
for  M&S 


Actions 

7.  Product 
development 
metamodel 

8.  Commercial 
SE  standards 

9.  Distributed 
simulation 
standards 

10.  DoDAF  utility 

-  DoDAF  2.0 
Acqn  Overlay 

-  Standards  for 
depiction  & 
interchange 

11.  Metadata 
template  for 
reusable 
resources 


Objective  3 


Improve 
model  and 
simulation 
capabilities 


Actions 

12.  Acquisition 
inputs  to  DoD 
M&S  priorities 

13.  Best  practices 
for  model/sim 
development 

14.  Distributed  LVC 
environments 

-  Standards 

-  Sim/lab/range 
compliance 

-  Event  services 

15.  Central  funding 
of  high-priority, 
broadly-needed 
models  &  sims 

-  Prioritized  needs 

-  Pilot  projects 

-  Expansion  as 
warranted 


Objective  4 


Improve 
model  and 
simulation 
use 


Actions 

16.  Help  defining 
M&S  strategy 

17.  M&S  planning 
&  employment 
best  practices 

18.  Foster  reuse 

-  Business  model 

-  Responsibilities 

-  Resource 
discovery 

19.  Info  availability 

-  Scenarios 

-  Systems 

-  Threats 

-  Environment 

20.  W&A 

-  Documentation 

-  Risk-based 

-  Examination 

21.  COTS  SE  tools 

22.  M&S  metrics 


Objective  5 


Shape  the 
workforce 


Actions 

23.  Definition  of 
required  M&S 
competencies 

24.  Harvesting  of 
commercial 
M&S  lessons 

25.  Assemble  Body 
of  Knowledge 
for  Acqn  M&S 

26.  M&S  education 
&  training 

-  DAU,  DAG  & 
on-line  CLMs 

-  Conferences, 
workshops  & 
assist  visits 

27.  MSIAC  utility 
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5  Acquisition  M&S  Objectives;  27  Actions 

(14  actions  broader  than  acquisition) 


Objective  1 


Provide 
necessary 
policy  and 
guidance 


Objective  2 


Enhance  the 
technical 
framework 
for  M&S 


Objective  3 


Improve 
model  and 
simulation 
capabilities 


Objective  4 


Improve 
model  and 
simulation 
use 


Actions 

1.  M&S 
management 

2.  Model-based 
systems 
engineering  & 
collaborative 
environments 

3.  M&S  in  testing 

4.  M&S  planning 
documentation 

5.  RFP  &  contract 
language 

6.  Security 
certification 


Key 

Broader  than  Acquisition 


Actions 

7.  Product 
development 
metamodel 

8.  Commercial 
SE  standards 

9.  Distributed 
simulation 
standards 

10.  DoDAF  utility 

-  DoDAF  2.0 
Acqn  Overlay 

-  Standards  for 
depiction  & 
interchange 

llJMetadata 
template  for 
reusable 
resources 


Actions 

12.  Acquisition 
inputs  to  DoD 
M&S  priorities 

13.  Best  practices 
for  model/sim 
development 

14.  Distributed  LVC 
environments 

-  Standards 

-  Sim/lab/range 
compliance 

-  Event  services 

15.  Central  funding 
of  high-priority, 
broadly-needed 
models  &  sims 

-  Prioritized  needs 

-  Pilot  projects 

-  Expansion  as 
warranted 


Actions 

16.  Help  defining 
M&S  strategy 

17.  M&S  planning 
&  employment 
best  practices 

18.  Foster  reuse 

-  Business  model 

-  Responsibilities 

-  Resource 
discovery 

19.  Info  availability 

-  Scenarios 

-  Systems 

-  Threats 

-  Environment 

20.  W&A 

-  Documentation 

-  Risk-based 

-  Examination 

21.  COTS  SE  tools 

22.  M&S  metrics 


Objective  5 


Shape  the 
workforce 


Actions 

23.  Definition  of 
required  M&S 
competencies 

24.  Harvesting  of 
commercial 
M&S  lessons 

25.  Assemble  Body 
of  Knowledge 
for  Acqn  M&S 

26.  M&S  education 
&  training 

-  DAU,  DAG  & 
on-line  CLMs 

-  Conferences, 
workshops  & 
assist  visits 

27.  MSIAC  utility 


5  Acquisition  M&S  Objectives;  27  Actions 

(10  actions  applicable  to  T&E) 


Objective  1 


Provide 
necessary 
policy  and 
guidance 


Objective  2 


Enhance  the 
technical 
framework 
for  M&S 


Objective  3 


Improve 
model  and 
simulation 
capabilities 


Objective  4 


Improve 
model  and 
simulation 
use 


Actions 

1.  M&S 
management 

2.  Model-based 
systems 
engineering  & 
collaborative 
environments 

3.  M&S  in  testing 

4^M&^plannin^ 

documentation 

5.  RFP  &  contract 
language 

6.  Security 
certification 


Key 

Broader  than  Acquisition 


T&E  Special  Interest 


Actions 

7.  Product 
development 
metamodel 

8.  Commercial 
SE  standards 

9.  Distributed 
simulation 

L  standards 

10.  DoDAF  utility 

-  DoDAF  2.0 
Acqn  Overlay 

-  Standards  for 
depiction  & 
interchange 

11.  Metadata 
template  for 
reusable 
resources 


Actions 

12.  Acquisition 
inputs  to  DoD 
M&S  priorities 

rl3.  Best  practices  ^ 
for  model/sim 
development 

^n^Mstrdmted^VC^ 

environments 

-  Standards 

-  Sim/lab/range 
compliance 

-  Event  services  J 

15.  Central  funding 
of  high-priority, 
broadly-needed 
models  &  sims 

-  Prioritized  needs 

-  Pilot  projects 

-  Expansion  as 
warranted 


Actions 

16.  Help  defining 
M&S  strategy 

17.  M&S  planning 
&  employment 
best  practices 


^8. 

Foster  reuse  ^ 

Business  model 

Responsibilities 

Resource 
discovery  j 

*19.  Info  availability' 

Scenarios 

Systems 

Threats 

v- 

Environment  j 

^0. 

W&A  > 

Documentation 

Risk-based 

Examination  J 

22.  M&S  metrics 


Objective  5 


Shape  the 
workforce 


Actions 

23.  Definition  of 
required  M&S 
competencies 


24.  Harvesting  of 
commercial 
M&S  lessons 


-  Conferences, 
workshops  & 
assist  visits 


27.  MSIAC  utility 


Summary  of  Actions  and  Products 
of  Interest  to  the  T&E  Community 


Establish  policy  and  guidance: 

>  To  require  documented  M&S  planning  as  part  of  the: 

■  Systems  Engineering  Plan 

■  T&E  Strategy  and 

■  T&E  Master  Plan 

>  For  security  certification  of  M&S  activities  falling  under  multiple 
Designated  Accreditation  Authorities  (DAA) 

Products:  Revised  policy  and  guidance  in  DoDI  5000.2,  8500.2; 
DAG;  &  TEMP  Planning  Guidance 


Summary  of  Actions  and  Products 
of  Interest  to  the  T&E  Community  (cont.) 


□  Foster  cost-effective  VV&A 

>  Develop  risk-based  methodology  and  associated  guidelines 
for  W&A  expenditures 

^Require  DoD-wide  standardized  documentation  of  VV&A 
^Examine  a  program’s  W&A  when  M&S  informs  major 
acquisition  decisions  and  unambiguously  state  the  purpose, 
key  assumptions  and  significant  limitations  of  each 
model/simulation  when  results  are  presented. 

Products:  Updated  DoDI  5000.61;  revised  policy  and 
guidance  in  DoDI  5000.2  &  DAG;  Guidance  &  training  for 
oversight  personnel 


Summary  of  Actions  and  Products 
of  Interest  to  the  T&E  Community  (cont.) 


□  Enable  readily-available  distributed  live-virtual-constructive 
environments 

>  Examine  opportunities  to  converge  distributed  simulation  standards 
(HLA,  DIS,  TENA,  ...) 

>  Establish  DoD-wide  standards  for  distributed  environments 

>  Foster  compliance  by  candidate  simulations,  labs,  and  ranges 

Products:  A  larger  collection  of  simulations,  labs,  &  ranges  ready  to 
be  employed  in  distributed  events 
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Summary  of  Actions  and  Products 
of  Interest  to  the  T&E  Community  (cont.) 


□  Educate  and  train  the  workforce  to  achieve  required  M&S  competencies 

>  Provide  M&S  knowledge  via  an  expanded  set  of  DAU  courses,  the  DAG,  and 
on-line  CLMs 

>  Provide  VV&A  training  for  oversight  personnel 

Products: 

>  Expanded  set  of  DAU  courses 

>  Improved  M&S  guidance  in  the  Defense  Acquisition  Guide 

>  On-line  Continuous  Learning  Modules 

■  “M&S  in  SE” 

■  “M&S  in  T&E” 
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Current  Status  and  Way  Ahead 


□  Completed  coordination  of  Draft  Acquisition  M&S 
Master  Plan  at  both  the  working  and  senior  levels 

□  Formal  DoD  coordination  pending  outcome  of 
Department  level  M&S  management  deliberations 

□  Re-chartering  the  AMSWG  with  new  responsibilities: 

>  Monitor  and  assist  with  AMSMP  execution 

>  Define  and  implement  a  process  to  identify  and  prioritize  M&S 
needs/projects  that  serve  a  broad  set  of  acquisition  users 

>  Advocate  Acquisition  M&S  needs  and  recommendations  at  the 
DoD  corporate  level 

>  Assess  the  state  of  acquisition  M&S,  relevant  new  technology, 
and  new  M&S  initiatives 

>  Interface  with  Industry  to  align  DoD  and  Industry  approaches  for 
M&S 
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Questions  /  Discussion  ? 
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Introduction 


Live-fire  testing  (Joint  Live  Fire  and  Title  10 
LFT&E)  is  now  in  its  third  decade. 

Contributions  of  LFT&E 

>  Identification  and  verification  of  system  and  munition 
design  strengths  and  weaknesses 

>  Quantification  of  crew  hazards  from  the  spectrum  of  insults 

>  Motivation  of  verification  &  validation  (V&V)  of  system-level 
analytical  models 

Key  to  success:  Identification  of  critical  evaluation  issues  —  significant 
design  and  deployment  concerns  —  on  which  to  focus  testing  and  analysis 
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Issues  Addressed 


>  Are  current  Live-Fire  Test  &  Evaluation  (LFT&E)  processes 
relevant  and  useful  in  light  of  current  changes  in: 

-  war-fighting? 

-  system  designs? 

-  integrated  information-centric  battlefield? 

-  extended  mission  timelines? 

>  What  changes  are  needed  in  LFT&E  strategies  to  address  the 
role  of  the  tested  platform  in  the  context  of  a  collection  of  assets 
available  to  the  unit  commander  to  prosecute  a  wide  range  of 
complex  missions? 


Issues  Addressed 


>  Considering  risk,  cost,  availability  of  hardware,  and  production 
schedule,  how  can  LFT&E  programs  be  constructed  to  ensure 
decision-makers  are  provided  with  data  in  a  format  that  allows 
them  to  ascertain  the  likelihood  of  completing  system  of  system 
(SoS)  collective  tasks? 

>  How  does  the  Missions  and  Means  Framework  (MMF)  provide 
a  foundation  for  the  development  of  cost-effective  LFT&E 
strategies  and  programs? 

>  What  issues  must  be  addressed  in  the  implementation  of  the 
proposed  MMF-based  SoS  task-focused  approach  to  LFT&E? 


Objective  of  Test  and  Evaluation 
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Assessing  Capabilities  Against 
Mission/Task  Requirements 


strategic  theater 


operational 


Does  the  mission  capability  package 

meet 

the  mission  capability  requirement? 


Modified  from  Tanenbaum,  Bray,  2005 


Acceptable 
Levels  of  Risk 
Failure? 


Alternative  SoS 
Actions? 

tactical 

/ 

C 


n 


1  platform/  1^ 

V/L 

[system  of  systems] 

Assessment 

Objectives  of  an  Effective 
LFT&E  Program 


An  effective  vulnerability  LFT&E  program  in  an  SoS  environment 
provides  data  that  allow  evaluators  to  assess: 

>  The  extent  to  which  the  platform  retains  those  capabilities 
determined  to  be  needed  for  completion  of  SoS  tasks,  when 
the  system  interacts  with  the  full  spectrum  of  ballistic  threats 
likely  to  be  encountered  in  combat. 

>  The  extent  to  which  the  SoS  is  able  to  complete  the  identified 
mission  tasks  in  the  joint  environment,  given  the  residual  and 
available  (as  determined  in  LFT&E)  capabilities  of  the  tested 
platform. 


Missions  and  Means  Framework: 
Foundation  for  Cost-Effective  LFT&E 


6.  Context,  Environment  (Military,  Civil,  Physical,  etc.) 

7.  OWNFOR  Why  =  Purpose,  Mission ^^^^^T^PFO^Wh^^Purpose^ission 


I 


The  MMF  provides  the  foundation  for  developing  cost-effective  LFT&E  strategies 
by  establishing  the  links  among  mission  tasks,  platform  capabilities,  platform 
components,  and  interactions  of  ballistic  threats  and  tested  platform. 


Modified  from  Sheehan,  et  al.,  2004 
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Critical  Issues  -  Keys  to  LFT&E  Success 


>  To  evaluate  armor  platform  vulnerability  solely  through  FUSL  testing,  tens  of 
thousands  of  FUSL  tests  would  be  required. 

-  Example  for  armor  platform: 

-- 10+  threat  classes  (small  arms,  mines,  etc;  several  variants  in  each) 

--  8+  attack  directions  per  threat 
--  500+  impact  points  per  attack  direction  ... 
easily  over  40,000  firings 

-  Fortunately  we  don’t  rely  only  on  testing;  many  combinations  can  easily  be  eliminated. 

--  Inspection  and  engineering  judgment 
--  Analysis 
-  Etc. 

-  Even  if  90%  can  be  easily  eliminated,  several  thousand  possible  firings  remain. 

>  A  typical  FUSL  LFT  can  afford  10-20  firings. 

-  Critical  issues  in  LFT&E  strategy  specify  rationale  for  which  10  or  20  shots  of  the 
possible  thousands  will  be  most  productive. 

--  Define  critical  evaluation  issues,  i.e.,  questions  to  be  answered  through  FUSL  LFT. 

--  Determine  how  supplementary  data  can  be  combined  with  FUSL  results. 


MMF  provides  basis  for  rational  cost-effective  strategy. 
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ASSESSMENT 

and  CONDUCT  STRATEGY 

EVALUATION  OF  TESTING  DEVELOPMENT 


Influence  of  MMF  on  LFT&E  Process 


Threat  Assessment 

System  Use  & 
Deployment 

System  Design 


*  Component 

*  Subsystem 

*  Surrogate 

» Ballistic  Hull 
&  Turret 


M&S 

Validation 

*  Evaluation 


I  I LFT  activities  in 
which  MMF 
plays  role 

I  I  LFT  activities  not 
affected  by  MMF 
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Differences  Between  Platform-Centric  and  MMF 
Task-Focused  Strategies 
-  Critical  Issues  - 


Critical  Issues 

Platform-centric  strategy 

1.  Vulnerability  of  the  platform? 

2.  Remaining  platform  mission  utility? 

3.  Role  of  BDAR  in  restoring  platform 
functional  capability? 

MMF-based  SoS  task-focused  strategy 

1 .  Reduction  in  the  ability  of  the  SoS  to 
prosecute  typical  missions? 

2.  Remaining  platform  capabilities? 

3.  Role  of  BDAR  and  other  maintenance 
actions  in  restoring  SoS  capabilities 
critical  to  mission  prosecution? 


[With  both  strategies,  causes  and  effects  of  crew  injury  are 
usually  critical  sub-issues.] 
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Differences  Between  Platform-Centric  and  MMF 
Task-Focused  Strategies 
-  Shotline  Selection  - 


E  I LFT  activities  in 
which  MMF  plays  role 
I  I  LFT  activities  not 
affected  by  MMF 


o 


o 

Z> 

Q 

O 

O 


<0 

LU 


•  Component 
’  Subsystem 

>  Surrogate 

>  Ballistic  Hull 
&  Turret 


1- 

LU  |i| 

o 

O  § 

H 

<  CO 

TJ  < 

2  CO 

E  ^ 

<  UJ 
o  (0 
(0 

$ 

< 

LU 

M&S 

Validation 

Evaluation 


Shotline  Selection 


Platform-centric  strategy 

Shotlines  are  selected  on  basis  of  technical 
risk  associated  with  inability  to  determine 

platform  capability. 

MMF-based  SoS  task-focused  strategy 

Shots  are  selected  on  basis  of  technical  risk 
associated  with  inability  to  determine 

effect  on  mission  prosecution  caused  by 
loss  of  platform  capabilities. 


(With  both  strategies,  shots  against  crew  are  selected  on 
basis  of  technical  risk  associated  with  ability  to 
predict  injury  to  crew  and  passengers.) 


12 


Differences  Between  Platform-Centric  and  MMF 
Task-Focused  LFT&E  Strategies 
-  Damage  Assessment  - 


I-  -1 LFT  activities  in 
which  MMF  plays  role 
I  I  LFT  activities  not 
affected  by  MMF 


Evaluation 


Damage  Assessment 
Platform-centric  strategy 

Platform:  Map  subsystem  loss  of  function  to 

combat  utility  via  03  4  mapping  construct 

Battle  Damage  Assessment  and  Repair: 
Expedient  repairs  to  restore  platform  to 

some  level  of  combat  utility 

MMF-based  SoS  task-focused  strategy 

Platform:  Map  subsystem  loss  of  function  to 

SoS  capabilities  by  analysis  and 
operational-type  tests 

Mission  Damage  Assessment  and  Repair: 

-  BDAR:  Expedient  repairs  to  restore 

some  platform  capabilities  immediately 
following  an  engagement 

-  Other  maintenance  procedures  to 
anticipate  future  mission  engagements 

(With  both  strategies,  crew  and  passenger  casualties  are 
assessed.)  13 


Building  a  Cost-Effective  LFT&E 

Program 
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Cost-Effective  LFT&E: 
Assessing  Vulnerability  Risk 


Likelihood  that  a  significant  vulnerability  (personnel  casualties,  catastrophic  loss  of 
system,  failure  to  complete  mission  tasks)  will  remain  undetected  in  a  fielded  platform 


Cost-Effective  LFT&E:  Weighing 
Costs  of  Vulnerability  Assessment 


FAILURE  TO  CONTROL  COSTS 


PREVENTION 


APPRAISAL/ 

DETECTION 


INTERNAL 

FAILURES 

A 

EXTERNAL 

FAILURES 

•RE-DESIGN 

•  SYSTEM  LOSS 

•RE-ENGINEERING 

•  PERSONNEL 

•RETROFIT 

CASUALTIES 

•  FAILURE  TO 
COMPLETE 
TASKS/ 
MISSIONS 
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Cost-Effective  LFT&E  Programs 

Assumptions 


>  Decomposition  of  relevant  missions  into  lower-level  tasks  will  have  been  completed. 

Missions  <  sto  accomplish 

comprise" - -  Tasks 


>  A  relationship  between  lower-level  tasks  and  the  minimum  levels  of  system 
capabilities  needed  to  complete  those  tasks  will  have  been  established. 

Tasks  < — .  to  perform 


enabled 
by 


Capabilities 


>  The  platforms  and  SoSs  that  provide  the  capabilities  to  complete  the  tasks 
will  have  been  identified;  redundancies  and  interdependencies  between  platforms 
are  known.  | _ 

Operational  Requirements  capture 
required  capabilities. 
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Cost-Effective  LFT&E:  Identify  Data 
Required  to  Assess  Vulnerability  in 
Ballistic  Interactions 


Links  (03  4)  between  levels  of 
capabilities  required  of  platform  to 
complete  tasks  (Level  3) 
and 

specific  mission  tasks  of  identified 
combat  scenarios  (Level  4) 


L4 


L2 


Platform-threat 
interactions  (LI) 


Links  (023)  between 

Links  (012)  between  platform- 

damage  state  vectors  (Level  2) 

threat  interactions  (Level  1) 

and 

and 

capability  state  vectors  (Level  3) 

damage  state  vectors  (Level  2) 
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Cost-Effective  LFT&E: 
Sources  for  Required  Data 


^  Results  of  material,  component,  subsystem,  &  system-level  tests  of: 

-earlier  models  or  current  model  of  system 

-earlier  or  contemporary  models  of  systems  with  similar  technologies 

^  Design  analyses  with  consideration  to  new  materials/technologies 

^  Combat  data  relevant  to  damage  mechanisms,  system  damage,  and  residual 
capabilities  of  system  as  associated  with  the  identified  threats 

^  Advanced  technology  and  concept  technology  demonstrations 

^  Force  development  tests/experiments 

^  Warfighting  experiments 

^  Engineering  analyses  and  controlled  damage  experiments 

^  Modeling  and  simulation  runs  that  incorporate  system  description,  threat 
characteristics,  and  damage-mechanisms  expected  in  threat-system  interactions 

^  Failure  Modes,  Effects,  and  Criticality  Analyses 

>  Results  of  developmental,  operational,  and  production  qualification  tests 
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Cost-Effective  LFT&E: 
Building  Program 


Identify  available  and  reliable  data. 
Identify  data  voids. 


Prioritize  data  voids  (use  experts  with  domain-specific 
experience,  Analytic  Hierarchy  Process,  Quality 
_ Function  Deployment,  etc.).  _ 


Identify  alternative  elements  in  a  LFT  program  to  fill 
data  voids  and  select  from  among  those  elements. 

Consider  data  priorities,  element  performance, 

costs,  &  risks. 
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Contribution  of  MMF  to  Planning 
for  LFT  and  Evaluation  of  Results 


The  ability  and  inability  to  obtain  reliable  data  pertaining  to 
specific  data  voids  have  consequences  for  specific  mission 
tasks,  as  shown  through  the  links  of: 
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Cost-Effective  LFT&E: 
Building  Program 


I  CDE 

\>  ^ 


Critical 

Critical 

Data  Void 

Data  Void 

X 

Y 

Critical  Data  Voids 


Capacity 

To 

address 
data  void? 


Time 
constraints 
of 

program 
.schedule?. 


Production 
schedule 
&  system 
hardware 
availability?'' 


Cost-Effective  LFT&E:  Measurin 
Costs  of  FUSL  LFT&E  Element 


FULL-UP 
SYSTEM-LEVEL 
LFT&E 


>  Identify  the  activities  and  sub-activities  of  the  LFT 
program  elements  for  which  costs  are  to  be 
measured. 

>  Apply  activity-based  costing  (ABC)  methodology  to 
measure  costs  of  activities/sub-activities  of  elements 
and  assist  decision-makers  in  addressing  questions: 

-  are  we  doing  the  right  things?  adding  value? 

-  are  we  doing  things  right?  Working  efficiently? 
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Cost-Effective  LFT&E:  Measuring 
Costs  of  FUSL  LFT&E  Element 


Applying  ABC:  Identify  the  direct  material  costs  and  the 
costs  of  resources  used  in  cross-functional  activities  of 
the  LFT  program  element.  _ 


COST  OF 

DIRECT 

ACTIVITIES 

MATERIALS 

Planning 

Test  Articles 

M&S 

Spare  Parts 

Test 

Munitions 

Performance 

Targets 

Documentation 

Evaluation 
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Cost-Effective  LFT&E:  Cost 

Complexities 


>  Costs  of  elements  of  vulnerability  assessment  are: 

-  incurred  by  a  multiple  number  of  players. 

-  reimbursed  from  a  multiple  number  of  sources. 

>  Methodologies  for  measuring  and  reporting  costs  across 
systems/divisions/Services  are  not  uniform. 

>  Cost  database  with  uniform  accounting  principles  is  not 
currently  available  for  purposes  of  cost  management/ 
control  and/or  projection  of  future  costs: 

-  to  afford  consistency  across  time  periods. 

-  to  afford  comparability  across  systems/Services. 
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Cost-Effective  LFT&E: 
Cost  Database  Requirements 


>  Identify  specific  data  to  be  collected  and  reported. 

>  Identify  methods  for  the  measurement  of  costs,  including  the  allocation  of  costs. 

>  Identify  procedures  for  disclosure  of  cost  measurement/allocation  methods 
if  alternative  methods  are  available. 

>  Identify  format  for  reporting  costs  (e.g.,  level  of  cost  aggregation). 

>  Identify  procedure  for  handling  missing  and  incomplete  data. 
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Cost-Effective  LFT&E: 
Implementation  Issues 


To  implement  a  cost-effective  MMF-based  SoS  task-focused 
approach  to  LFT&E,  need: 

>  Integration  under  the  MMF  of  the  efforts  of  acquisition, 
requirements,  M&S,  T&E,  and  training  communities,  achieved 
only  through  the  support  of  top  levels  of  defense  administration 

>  Allocation  of  resources  to  appropriate  Service  divisions  to 
ensure  availability  of  test  assets,  including  hardware,  testing 
facilities/ranges,  and  people  with  the  levels  of  expertise  needed 
for  the  planning  and  evaluation  processes 
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Cost-Effective  LFT&E: 
Implementation  Issues 


To  implement  a  cost-effective  MMF-based  SoS  task-focused 
approach  to  LFT&E  need: 

>  Construction  of  platform  operational  requirements  based  on  the 
capabilities  needed  for  the  completion  of  multiple  tasks  of 
multiple  missions 

>  Identification  and  measurement  of  costs  of  LFT&E  elements 
according  to  a  consistent  methodology  to  allow  a  weighing  of 
costs  against  the  value  added  in  conducting  individual  program 
elements 
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“Mo  More  Second  Opinions:  Organizing 
<Do(D  ModS  Within  the  Acquisition 

CycCe  ” 


James  (F.  O  ’(Bryon 
The  O  X Btyon  Cjroup 

443-528-2711 

NDIA  T&E  Conference, 
Jacksonville,  FL 

March  9,  2006 


M&S  AND  T&E 
ARE  PARTNERS, 
NOT  COMPETITORS 


MODELING  AND  SIMULATION 

IN 

TEST  6  EVALUATION 


Modeling  and  simulation  are  an  integral 
part  of  T&E  and  not  to  be  looked  at  as  a 
substitute,  nor  a  means  to  save  money. 
M&S  and  testing  are  mutually 
supportive  and  none  is  complete 
without  the  other. 


M  &  S  PLAY  A  VITAL  ROLE  EARLY  ON 
IN  SYSTEM  DESIGN  AND  VERIFICATION 

Source:  R.  Garrett,  “Opportunities  in  Modeling  and  simulation  to  Enable  Dramatic 
Improvements  in  Ordnance  Design,  “presented  to  the  Committee  on  Bridging  Design  and  Manufacturing. 
National  Research  Council,  Washington,  DC.,  April  29,  2003. 


SYSTEM  LIFE  CYCLE 


A  Proposal  That  Might  Work 
Program  Manager  Magazine 


JAMES  F.  O'BRYON 


In  the  following  few  pages,  I  discuss 
my  personal  thoughts  on  an  issue 
of  paramount  importance  not  only 
to  the  Department  of  Defense,  but 
also  to  the  nation’s  defense.  My 
hope  is  that  this  article  will  provoke  se¬ 
rious  thought  and  meaningful  action  to 
resolve  the  issues  raised. 

A  Lock  Bade 

Since  arriving  in  the  Pentagon  just  over 
12  years  ago,  and  for  more  than  a  decade 
before  that  serving  as  a  weapons  analyst 
in  the  Department  of  Defense  (DoD)  in¬ 
frastructure  away  from  the  Washington 
area,  I  have  been  witness  to  numerous 
and  surprisingly  similar  technical  and 
management  discussions  about  the  need 
to  get  the  modeling  and  simulation  ca¬ 
pabilities  of  the  DoD  organized,  incen- 
tivized,  under  control,  and  more  efficient 
to  better  serve  the  weapons  development 
and  acquisition  process. 

These  discussions  included  such  issues 
as  a  common  and  meaningful  model  ar¬ 
chitecture,  model  inter-connectivity,  lan¬ 
guage  consistency,  validation,  model 
proliferation,  and  configuration  control. 
They’ve  also  covered  the  problems  of  du¬ 
plication,  modeling  “stovepipes,”  the  lack 
of  meaningful  and  up-to-date  docu¬ 
mentation  supporting  M&rS,  and  of 
course,  the  lack  of  model  realism. 


O'Bryan  serves  as  the  Deputy  Director,  Operational 
lest  and  Evaluation,  Live  Fire  Testing,  in  the  Office 
of  the  Secretary  of  Defense,  The  Pentagon.  Wash¬ 
ington,  DC.  His  undergraduate  degree  is  in  Mathe¬ 
matics,  and  he  also  holds  two  graduate  degrees: 
one  in  Operations  Research  from  The  George 
Washington  University  and  another  through  the 
Electrical  Engineering  Department  of  the  Massa¬ 
chusetts  Institute  of  Technology 


TRUTH  #  1 


TRUTH  #  2 


“PMs  &  PEOs 
Control 
Largest 
Funding 
Blocks” 


TRUTH  #  3 


TRUTH  #  4 


“PMs  Have 
Short  Time 
Horizons 
And,  Hence, 
On  Their 
Investment, 
Decisions” 


TRUTH  #  5 


“Realistic 
M&S  Is  Not 
Necessarily 
Viewed  As  A 
Benefit  By 


The 


TRUTH  #  6 


“The  Golden 
Rule:  Them 
That  Have 
The  Gold, 
Make  The 
Rules” 


“MASTER” 


MODELING  AND 
SIMULATION  TEST  AND 
EVALUATION  REFORM 


What  is  ‘MASTER’? 


•  MASTER  is  a  management  approach  to 
modeling  and  simulation  in  support  to  the 
defense  department’s  policy  of  simulation- 
based  acquisition 

•  It  will  provide 

•  critical-mass  funding 

•  add  discipline  to  the  development  of 
modeling  and  simulation 

•  assure  that  funds  expended  on  modeling  and 
simulation  are  spent  to  further  the  state  of 
the  artj  including  VV&A 

•  add  connectivity  across  various  model 
vectors  being  developed 

•  free  up  the  Program  Manager’s  time  & 

concerns  about  modeling  and  simulation 
support 

•  assure  the  most  realistic  models  & 
simulations  are  exercised  in  designing 

testing,  evaluating,  training,  fielding  and 
fighting  our  systems. 


Consortium  Discussion 


Program  Managers  would  initially  describe  their 
system(s),  acquisition  strategy,  and  M&S 
requirements  to  a  consortium  which  would  then 
parse  out  these  needs  into  vectors  of  M&S  technical 
responsibility. 

Consortium  Members,  who  are  charged  with  having 
knowledge  of  state  of  the  art ,  as  well  as  where  it 
exists  within  and  outside  of  their  respective 
organizations,  would  make  the  decisions  as  to  which 
M&S  tools  best  suit  the  PM’s  needs  and  where  the 
funds  would  be  expended  to  meet  the  specific 
requirements  of  each  Program  Manager’s  system(s). 
They  would  upgrade  extant  models  where  available 
and  originate  M&S  only  when  absolutely  necessary. 
In  many  instances,  these  investments  would  be 
allocated  to  organizations  external  to  the 
Consortium  Membership  itself. 


Must  Have  Up-Front 
Investment  in  M&S!! 


“I  expect  programs  to  make  the  up-front 
investment  in  modeling  and  simulation 
application  technology,  and  will  be 
looking  for  evidence  of  that  investment 
in  program  planning  and  execution.” 


Honorable  Jacques  S.  Gansler,  Under  Secretary 
of  Defense  (Acquisition  and  Technology)  1998 


If  there’s  no  new  money,  where 
will  the  money  come  from  to 
fund  this  “MASTER”  initiative? 


Where  Would  Money  to  Fund  the 
Consortium  Come  From? 
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A  modest  tax  (“greens  fee”)  would  be  assessed  upon 
every  Program  Manager’s  total  budget.  These  funds 
would  be  placed  in  the  Consortium’s  account  to 
provide  the  needed  M&S  support  to  the  Program 
Manager. 

The  proposed  “tax”  would  be  a  percentage  of  the 
Program  Manager’s  budget  (perhaps  2-3%).  This  is 
significantly  less  than  what  is  currently  spent  by 
PM’s  on  a  plethora  of  isolated  M&S  activities. 

The  tax  would  not  be  at  the  discretion  of  the 
Program.  It  would  be  a  policy  decision  and 
implemented  early  on  at  the  OSD  Comptroller  level. 

Funds  would  be  removed  early  to: 

•  enable  sufficient  time  to  develop  the  needed  M&S, 
and, 

•  avoid  the  tendency  to  cut  the  funding  of  modeling 
and  testing  programs,  when  problems  arise  and 
budgets  get  tight 


Potential  Modeling  Vectors 
Needed  for  the  Testing  and 
Training  Communities 


SOME  EXAMPLES: 

TERRAIN 

WEATHER 

CADCAM  SYSTEM  DESCR 
AERODYNAMIC  FLOW/FLIGHT 
STABILITY 
6  DOF  FLY-OUT 
TARGET  SIGNATURES 
SENSOR/FUZING 
SMOKE/OBSCURATION 
C3I 
EW 

ACOUSTIC 
BALLISTIC 
1-1  ENGAGEMENT 
MxN ENGAGEMENT 
VULNERABILITY 
LOGISTICS 
MANY  OTHERS 


What  are  Some  of  the  Benefits 
of  Forming  a  Consortium  to 
Oversee  M&S  Investment 
Within  the  DoD? 


1.  It  would  assure  that  Program  Managers 

have  the  best  and  most  realistic  model 
support  for  their  programs. 

2.  It  would  establish  the  necessary  consortium 

protocols  for  model  architecture, 
languages  and  protocols,  insuring  that  no 
funds  would  be  invested  in  model 
development  or  upgrades  unless  they 
meet  these  protocols,  thereby  facilitating 
interoperability. 

3.  It  would  assure  that  model  investments 

would  be  directed  toward  extending  the 
capability  of  extant  models  and 
simulations  rather  than  spending 
significant  funds  reinventing  and  re¬ 
buying  codes  which  exist  or  exist  in 
part. 


Consortium  Benefits? 
(Continued) 


4.  The  structure  would  provide  an  adequate 

source  of  funding  to  extend  the  state  of  the 
art  in  the  M&S  base,  instead  of  being  at 
the  whim  of  the  Program  Manager, 
typically  trying  to  maximize  the  short¬ 
term  return. 

5.  It  would  focus  national  expertise  in  each 

technical  discipline  to  assure  that 
decisions  on  which  model  investments 
were  indeed  needed  in  each  of  these 
disciplines. 

6.  It  would  free  up  the  Program  Manager’s 

time  and  attention  to  other  management 
responsibilities  and  allow  the  Consortium 
to  provide  the  needed  M&S  support  for 
each  respective  program. 
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A  PUBLICATION  OF  THE  DEFENSE  ACQUISITION  UNIVERSITY! 


4  DoD’s  Technical 
Information  Broker 


Defense  AT&L  Interviews 
R.  Paul  Ryan,  Administrator 
mmmm  Defense  Technical 
Information  Center 
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ALSO 


]  America’s  Imprudent 
4Vi  and  Unsustainable 

c?|2lE?r‘  i  ii .  Fiscal  Path 


David  M.  Walker, 
U.S.  Comptroller  General 


dig  And  Introducing 

Defense  AT&L’s  First  Graphic  Article 


DoD’s  Modeling  and  Simulation 
Reform  in  Support  of  Acquisitior 

Stop  Kicking  the  M&S  Can  Down  the  Road 


James  F.  O’Bryon 


as  being  among  its 
primary  methods  for 
reducing  time  to  mar¬ 
ket  for  defense  systems 
and  reducing  the  cost  of  these  sys 
terns  at  the  same  time.  The  fol¬ 
lowing  statement  is  contained 
in  a  letter  dated  March  21, 

2000,  addressed  to  the  Office 
of  the  Secretary  of  Defense, 

Service  secretaries,  the  De¬ 
fense  Intelligence  Agency, 
and  the  Joint  Chiefs  of  Staff; 
it  is  cosigned  by  the  under 
secretary  of  defense  (acqui¬ 
sition,  technology  and  logis¬ 
tics)  (USD(AT&L))  and  the  di¬ 
rector,  operational  test  and 
evaluation,  (DOT&E):  “We  have 

the  funds,  however,  are  spent  in  support  of  the  r 
development,  test,  and  evaluation  of  new  def 
quisition  programs. 

In  an  article  in  the  July  2005  issue  of  National 
Magazine,  David  W.  Duma,  the  Pentagon’s  actii 
tor,  operational  test  and  evaluation,  wrote  that 
fense  Department  needs  to  better  manage  its  sir 
programs.  I  think  we’ve  kind  of  lost  our  way 
partment  with  modeling  and  simulation.  Multij 
cies  are  buying  duplicate  technologies,  rather  th 
dinating  efforts.  We  are  using  more  model 
simulation.  But  it’s  not  focused,  it’s  scattered.  Ev 
is  building  their  own.” 

Not  a  New  Problem 

1  couldn’t  agree  more.  So  why  does  the  DoD  cor 
lose  its  way  using  more  M&S  but  in  a  “scattered 


stressed  that  we  must  make  better  use  of  modeling 
and  simulation  (M&S)  to  improve  the  acquisition 
process,  reduce  costs,  enhance  T&E  [test  and  evalua¬ 
tion],  and  shorten  development  times  for  our  new  sys¬ 
tems.  We  are  convinced  that  efficient  use  of  M&S 
throughout  the  system  life  cycle  will  net  great  dividends 
in  efficiencies.” 

Few  people  would  argue  that  M&S  is  not  an  important  el¬ 
ement  in  the  acquisition  process.  The  question  is  this: 
Has  there  been  progress  within  DoD  to  efficiently  orga¬ 
nize,  fund,  develop,  promulgate,  and  maintain  configu¬ 
ration  control  of  the  DoD’s  massive  and  diverse  M&S  ac¬ 
tivities  to  yield  the  efficiencies  so  clearly  stated  in  the 
letter  quoted  above?  Estimates  for  how  much  is  spent  an¬ 
nually  on  M&S  in  the  DoD  range  from  $5  billion  to  $30 
billion,  depending  on  how  one  defines  M&S.  Some  of  this 
is  spent  on  M&S  in  support  of  training.  The  majority  of 


odeling  and  simula¬ 
tion— M&S— has  long 
been  touted  by  the  De¬ 
partment  of 
Defense 


O’Bryon  served  as  deputy  director,  operational  test  and  evaluation  in  the  Office  of  the  Secretaiy  of  Defense  until  November  2001.  He  current 
as  a  consultant  to  ORSA  Corporation.  Aberdeen.  Md. 
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IF  YOU  HAVE  SOME  IDEAS  YOU'D  LIKE 
TO  SHARE  OR  WOULD  LIKE  TO 
CHALLENGE  SOME  O  THESE  IDEAS,  I 
WOULD  WELCOME  YOUR  IDEAS. 
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THE  O'BRYON  GROUP 

Consultants 

National  Defense  &  Security 

JAMES  F.  O'BRYON 


1608  South  Tollgate  Road  Voice  (410)  515-03 

Bel  Air,  MD  21015-5825  Fax  (410)  569-96 

iamesobrvon@obrvonaroLto.com  Cell  (4431 528-27 

Call  me  at  410-515-0345  op  email  me  at 

jamesobryon@obryon 
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TECHNOLOGIES 


Modeling  and  Simulation  of 
System  of  Systems 
The  History  of  the  LPD  17  PRA 

Testbed 


M&S  Studies  in  the  Context  of  T&E  and  Acquisition 


LPD  17  PRA  Testbed 
Vincent  M.  Ortiz 
AVW  Technologies 
8  March,  2006 


OVERVIEW 


AVW 

TECHNOLOGIES 


•  LPD  17  San  Antonio  Ship  Class 

•  LPD  17  Probability  of  Raid  Annihilation  (Pra) 
Testbed  Description  and  Architecture 

•  Testbed  Development  Organization 

•  Testbed  Documents 

•  Testbed  Requirements 

•  Testbed  Development  Process 

•  Testbed  Spiral  Development  and  Schedule 

•  Testbed  Analysis  and  Scenario  Development 


AVW 

TECHNOLOGIES 


LPD  17  CAPABILITIES 


*  The  LPD  17  capabilities  include: 

-  State-of-the-art  command  and  control  suite 

-  Advanced  ship  survivability  features  that  enhance  its  ability 
to  operate  in  the  unforgiving  littoral  environment  (low  radar 
cross  section) 

-  Substantially  increased  landing  force  vehicle  lift  capacity 
(23,600  square  feet  of  vehicle  storage  space), 

-  Large  flight  deck  (land  2  MV-22  or  4  CH-46)  and  well  deck 
(holds  2  Landing  Craft  Air  Cushion  {LCAC}) 

*  The  LPD  17  is  the  first  amphibious  ship  designed  to 
accommodate  the  Marine  Corps’  “mobility  triad” 

-  Expeditionary  Fighting  Vehicle  (EFV) 

-  LCAC 

-  MV-22  Osprey  tilt  rotor  aircraft. 


OUR  FOCUS  WILL  BE  ON  THE  COMBAT  SYSTEM 


Low  Radar  Cross  Section  (RCS) 


SPS-48E  3D  Air  Search  Radar 


SPQ-9B  Air/Surface  Tracking  Radar 


SSDS  Control  and  Decision  System 


Nulka  Active  Decoy 


SLQ  32A(V)2  EW  System 


RAM  Missile  System 


BACKGROUND  -  Pra 


AVW 

TECHNOLOGIES 


OBJECTIVE:  ASSESS  LPD  17’s  PRA 
(ABILITY  TO  DEFEND  ITSELF  AGAINST  INCOMING  MISSILES) 

■  CNO’s  Anti-Air  Warfare  Capstone  Requirements  Document  mandated  the  ship 
self  defense  capability  for  specific  ship  classes  and  established  the  Probability  of 
Raid  Annihilation  (PRA)  as  the  primary  Measure  of  Effectiveness  (MOE)  to  assess 
ship  combat  system  suites. 

■  PRA  is  defined  as  the  ability  of  a  particular  stand-alone  ship,  as  an  integrated 
system,  to  detect,  control,  engage,  and  defeat  a  specified  raid  of  anti-ship  cruise 
missile  (ASCM)  threats  with  a  specified  level  of  probability  in  the  operational 
environment. 


■  Th®  PRA  MOE  is  a  system -of -systems  measure  which  is  levied  on  the  ship 
defense  suite  as  a  whole  to  properly  detect,  control,  and  engage  (annihilate)  a 
raid  of  incoming  threat  ASCMs.  Thus,  it  doesn’t  measure  the  performance  of  any 
particular  ship  defense  element;  rather  it  measures  the  system  performance  of  all 
the  ship  defense  elements  across  the  complete  battle  timeline. 

■  The  LPD  17  class  is  the  first  U.S.  Naval  ship  class  required  to  demonstrate  its 
ability  to  defeat  specific  anti-ship  cruise  missile  threats  to  achieve  a  statistical 
Pra- 
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NAVY’S  SOLUTION  TO  Pra 
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Pra  Assessment  is  a 
Three  Pronged  Approach 
-  Test  against  actual  ship  (LPD  17) 

•  Pro  -  Test  Drones  Against  the  Actual  Ship 

•  Con  -  Limited  Firing  Events, 

Cannot  Fire  ASCM  Against  Manned  Ship 


-  Test  against  Self  Defense  Test  Ship  (SDTS) 

•  Pro  -  Can  Fire  ASCM  Against  SDTS 

•  Con  -  Limited  Representation  of  the  Actual  Ship, 

Limited  Firing  Events 

-  Test  using  M&S  (LPD  17  Testbed) 

•  Pro  -  Can  Runs  Numerous  Different  Scenarios,  Events 

•  Con  -  Developmental  Cost,  Limiting  Assumptions 
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LPD  17  PRA  TESTBED 


Naval  ResearcfrLab 
Washington,  DC  \ 


Scenario  & 
Environment 
data  drivers 


SIMDIS 


JDARS 


SSDS 


CEP 


RAM 

M-on-N 


RAM 

launcher 


Decoys 


JHU  Applied  Physics  Lab 
Laurel,  MD 


NAWC  Weapons  Division 
China  Lake 


Threats/targets  I  SLq.32  »  SPS-48E  I  SPQ-9B 
seekers  &  airframes  I  w  0! 


Network  Interface  Layer 
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Geographically  Distributed  Federation  of  Tactical  HWIL, 
Tactical  SWIL  and  Digital  Physics  Based  Models 


OVERVIEW 
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•  LPD  17  San  Antonio  Ship  Class 

•  LPD  17  Probability  of  Raid  Annihilation  (Pra) 
Testbed  Description  and  Architecture 

•  Testbed  Development  Organization 

•  Testbed  Documents 

•  Testbed  Requirements 

•  Testbed  Development  Process 

•  Testbed  Spiral  Development  and  Schedule 

•  Testbed  Analysis  and  Scenario  Development 
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TECHNOLOGIES 


LPD  17  Pra  TESTBED  OVERVIEW 


MANAGEMENT  APPROACH: 


Organization 

Meetings 

Documents 

Schedule 


BOUND  THE  PROBLEM: 

Testbed  Requirements 
Fidelity 

Ship  Configuration 
Environment 
Threat  Types 
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TECHNICAL  APPROACH: 


Physics  -  Based 
Non  -  Real  Time 
Distributed,  RTI  Solution 
HLA  Compliant 
Spiral  Development 


BOUND  THE  ANALYSIS: 

Finite  Number  of  Runs 
(Geographic  Location 
Ship  Configuration 
Season,  Time  of  Day 
Threat  Types) 
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ROLES  &  RESPONSIBILITIES 


PMS  317 

■  Manages  Funding 

■  Drives  Schedule 

■  V&V  Manager 

■  DT  Accrediting  Authority 

PEO  IWS  CSE 

■  Manages  Testbed  Design 
and  Development 

NRL 

■  Testbed  Integrator 

NSWC  Corona 

■  Test  Resource,  Planning 
and  Data  Collection 
Agent 


Element  PMs 

■  Co-Chair  SCP. 

■  Review  &  Approve  SOWs 
associated  with  M&S 
Development. 

■  Manage/  Participate  in 
Model  Development. 

■  Responsible  for  the 
Credibility  of  their 
Respective  Models 

Model  Developers 

■  Develops/  Integrates 
Models 

COMOPTEVFOR 

■  Participates  as  the  OT 
Accrediting  Authority 
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LPD  17  PRA  ORGANIZATION 


MANAGEMENT  IPT 

*  LPD  17  Combat  System  Integration  Manager  *  Ship  Self  Defense  Combat  Systems  Engineer 

•  LPD  17  Test  Director  •  Deputy  SSD  CSE 


WORKING  IPT 


CS  Element  PMs 


SIMULATION  CONTROL  PANELS  (SCP) 


Threat 

! 

Softkill 

■ 

Radars 

Natural 

Environment 

1 _ 

Hardkill 

1 _ 

CEC 

■ _ 

M&S  Developers 

4 


Scenario 


Test  Bed 


SSDS 


OVERVIEW 
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•  LPD  17  San  Antonio  Ship  Class 

•  LPD  17  Probability  of  Raid  Annihilation  (Pra) 
Testbed  Description  and  Architecture 

•  Testbed  Development  Organization 

•  Testbed  Documents 

•  Testbed  Requirements 

•  Testbed  Development  Process 

•  Testbed  Spiral  Development  and  Schedule 

•  Testbed  Analysis  and  Scenario  Development 


AVW  Process  developed  the  Approach,  Requirements  and  Build  Plan 
AVW  Database  Produced  the  Requirements  and  VV&A  Documents 
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TESTBED  SCHEDULE 


OVERVIEW 
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•  LPD  17  San  Antonio  Ship  Class 

•  LPD  17  Probability  of  Raid  Annihilation  (Pra) 
Testbed  Description  and  Architecture 

•  Testbed  Development  Organization 

•  Testbed  Documents 

•  Testbed  Requirements 

•  Testbed  Development  Process 

•  Testbed  Spiral  Development  and  Schedule 

•  Testbed  Analysis  and  Scenario  Development 


OVERVIEW 
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•  LPD  17  San  Antonio  Ship  Class 

•  LPD  17  Probability  of  Raid  Annihilation  (Pra) 
Testbed  Description  and  Architecture 

•  Testbed  Development  Organization 

•  Testbed  Documents 

•  Testbed  Requirements 

•  Testbed  Development  Process 

•  Testbed  Spiral  Development  and  Schedule 

•  Testbed  Analysis  and  Scenario  Development 
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LPD  17  PRA  TESTBED 


Naval  ResearcfrLab 
Washington,  DC  \ 


Scenario  & 
Environment 
data  drivers 


SIMDIS 


JDARS 


SSDS 


CEP 


RAM 

M-on-N 


RAM 

launcher 


Decoys 


JHU  Applied  Physics  Lab 
Laurel,  MD 


NAWC  Weapons  Division 
China  Lake 


Threats/targets  I  SLq.32  »  SPS-48E  I  SPQ-9B 
seekers  &  airframes  I  w  0! 


Network  Interface  Layer 


g 


Geographically  Distributed  Federation  of  Tactical  HWIL, 
Tactical  SWIL  and  Digital  Physics  Based  Models 


AVW  TESTBED  SPIRAL  DEVELOPMENT 
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n 

i 

x 

Develop  Model 
Build/  CM  Plan  and 
V&V  Plan 


Develop  the  Models  and  Testbed 
Through  Spiral  Development 
Of  Four  Successive  Builds 


Submit: 
Final  Report, 
Accreditation 
Package 


DEVELOPMENT  TIMELINE 


Build 

1 
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Time 


OVERVIEW 
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•  LPD  17  San  Antonio  Ship  Class 

•  LPD  17  Probability  of  Raid  Annihilation  (Pra) 
Testbed  Description  and  Architecture 

•  Testbed  Development  Organization 

•  Testbed  Documents 

•  Testbed  Requirements 

•  Testbed  Development  Process 

•  Testbed  Spiral  Development  and  Schedule 

•  Testbed  Analysis  and  Scenario  Development 
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ANALYSIS  OVERVIEW 


•  Objective  -  to  bound  a  problem  having  infinite  possibilities. 

•  Number  of  variables  limited  by  time  to  perform  analysis  runs 

•  Make  analysis  space  finite  and  within  a  reasonable  operational 
context. 

•  Not  skewed  in  any  one  point  of  view. 

•  Scientifically  supported;  no  need  spending  money  on  physics 
excursions. 

•  Approach  should  be  consistent  across  ship  classes. 

•  The  scenarios  are  scripted  to  prevent  other  ships,  LCACs,  and 
aircraft  from  interfering  with  the  engagement  sequence. 

•  The  threat  should  not  be  distracted  from  its  target  by  these 
other  units  in  the  scenario. 


AVW 

TECHNOLOGIES 


SCENARIO  OVERVIEW 


•  Background 

-  LPD  17  is  part  of  an  Expeditionary  Strike  Group  (ESG). 

-  Early  detection  and  engagement  of  the  launching  platform  is  not 
in  the  context  of  the  Pra  Assessment  since  its  focus  is  ASCM  self 
defense. 

-  Battle  force  networking,  force  link  tracking  and  force  cooperative 
engagements  standpoint  are  not  supported  by  the  LPD  17  Pra  self 
defense  context;  therefore,  the  ship  will  be  in  a  worst  case 
situation  with  no  data  links  active,  requiring  it  to  perform  the  entire 
detect-to-engage  sequence  on  its  own. 

•  Initial  Conditions 

-  Detailed  geometries,  tactics,  and  operational  situations  will  be 
developed  to  provide  boundary/initial  conditions  for  each  run  as 
well  as  to  drive  the  variables  during  the  run  to  ensure  operationally 
realistic  and  consistent  runs  for  analysis. 
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SCENARIO -THREATS 


•  Combat  System  Setup 

-  Conducting  wartime  transit  steaming  and  mission  operations  with 
the  ship’s  defensive  systems  set  up  to  counter  ASCM  threats 
automatically. 

-  No  operator  actions  required  except  for  NULKA  launches,  which 
will  be  treated  as  a  time  delay  in  the  automatic  engagement 
sequence  based  on  nominal  operator  reaction  times. 

-  Surface  Warfare  Development  Group  (SWDG)  Tactical 
Memorandums  (TACMEMOs)  and  other  tactics  and  doctrine 
publications  will  be  used  to  configure  the  LPD  17  Combat  System 
representation  in  the  Testbed  for  execution  of  each  simulated 
engagement  run. 

•  Threats 

-  Threats  will  be  fired  in  stream  raids  of  x  sec  spacing,  from  eight 
true  bearings  (±8°)  about  the  compass  rose. 

-  Intelligence  on  threats  and  threat  tactics  will  be  used  to  develop 
detailed  threat  engagement  scenarios  for  each  run. 

-  A  set  of  5  representative  threats  were  selected. 
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SCENARIO  -  SHIP  CONFIG 


■  Case  1  -  “Clean”  RCS 

■  The  lowest  possible  realistic  RCS  and  IR  values 
representative  of  the  ship  in  transit  condition  in  wartime. 

■  The  flight  deck  will  be  as  free  of  aircraft  and  yellow  gear  as 
possible. 

■  The  stern  gate  will  be  closed. 

■  Case  2  -  “Dirty”  RCS 

■  Near  worst  possible  realistic  RCS  and  IR  values 
representative  of  the  ship  in  an  operational  environment 
conducting  well  deck  and  aviation  operations. 

■  SH-60s  or  MV-22s  (whichever  has  higher  RCS  value)  will  be 
chained  on  the  deck. 

■  Stern  gate  will  also  be  open  with  the  well  deck  empty  of 
LCACs  and  water. 
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SCENARIO  -  ENVIRONMENT 


•  Approach 

-  Provide  representative  sample  space  of  environmental  and  other 
variables  such  as  water  vapor,  specific  humidity,  particulates, 
temperature,  air  pressure,  ducting,  sea  state  3  and  associated  wind 
direction,  wind  speed,  wave  height  and  wave  direction,  as  well  as 
sun  angle. 

•  Season 

-  Summer  Scenario 

-  Winter  Scenario 

•  Time  of  Day 

-  Shortly  after  Sunrise 

-  Noon 

-  Afternoon 

-  Shortly  before  Sunset 

-  Midnight 

•  Sea  State 

-  Established  as  Sea  State  3  for  All  Scenarios 
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SCENARIO 


Banghazr 


jtta 


Geography  1 
Open  Ocean  -  Mid-Med 


GEOGRAPHIES 


Geography  2 
Straits  of  Hormuz 
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Threat  1 
T  2 
T  3 
T  4 
T  5 


SCENARIO  VARIABLES 


Med  SOH 

Geographic  Location 

Clean  RCS  Ship 
Dirty  RCS  Signature 


Time: 

Sunrise 

Noon 

Afternoon 

Sunset 

Midnight 


8  Threat 
Radials 

/ 

/ 

/ 

A  /  Season 
A/  Winter 
Summer 
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ANALYSIS  APPROACH 


2  Geographies 

-  Med  Open  Ocean 

-  Straits  of  Hormuz 

-  Provides  Stressing  and 
Non-Stressing  Locations 

2  Environments 


2  Radar  Cross  Sections 

-  Clean,  Minimized  RCS 

-  Dirty,  Open  Well,  Helo  on 
Deck 

-  Provides  Large  and  Small 
Signatures 

5  Threats 


-  2  Times  of  Year 

-  5  Times  of  Day 

-  No  Rain 

-  Provides  Nominal 
Changes  in  Environment 


-  T1R1,  T2,  T3,  T5,  T7 

-  8  Threat  Bearings 

-  45  Deg  Intervals 

-  Provides  Combat  System 
Performance  from  all 
Directions 


PERFORM  ONE  RUN  FOR  EACH  COMBINATION  OF  6  VARIABLES 
STATISTICALLY  A  REPRENTATIVE  SAMPLING  THROUGH  THE  SPACE 
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Threat  1 
T  2 
T  3 
T  4 
T  5 


TESTBED  SAMPLE  SPACE 


PRA  Event 

Med  SOH 

Geographic  Location 


PRA  Event  =  20 

(5  Threat  x  2  GEO  x  2  Signature) 
PRA  Run  =  80 

(5  Time  x  8  Radials  x  2  Seasons) 
One  Firing  for  Each  Unique  Run 
Total  Number  of  Runs  =  1600 


Clean  RCS  Ship 
Dirty  RCS  Signature 


PRA  Run 


Time: 

Sunrise 

Noon 

Afternoon 

Sunset 

Midnight 


8  Threat 
Radials 

PRA  Event 


Season: 
//  Winter 
Summer 
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Threat  1 
T  2 
T  3 
T  4 
T  5 


TESTBED  PRA  CALCULATIONS 


PRA  (Event)  = 
PRA  (Threat)  = 


PRA  Event 

Med  SOH 

Geographic  Location 


#  Successes  (20  PRA  Values) 
80 

I  PRA  Events  (5  PRA  Values) 
4 

PRA  (Geography)  =  I  PRA  Events  (2  PRA  Values) 

10 

PRA  (Ship  Sig)  =  I  PRA  Events  (2  PRA  Values) 

10 

PRA  Overall  =  I  PRA  All  Event  (1  PRA  Value) 


PRA  Event  =  20 

(5  Threat  x  2  GEO  x  2  Signature) 
PRA  Run  =  80 

(5  Time  x  8  Radials  x  2  Seasons) 
One  Firing  for  Each  Unique  Run 
Total  Number  of  Runs  =  1600 


Clean  RCS  Ship 
Dirty  RCS  Signature 


PRA  Run 


Time: 

Sunrise 

Noon 

Afternoon 

Sunset 

Midnight 


8  Threat 
Radials 

PRA  Event 


Season: 
//  Winter 
Summer 
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TESTBED  SCHEDULE 


ID 

Task  Name 

Start 

Finish 

1 

Model  Build  1 

Mon  10/25/04 

Mon  1 0/25/04 

2 

Testbed  Build  1 

Tue  10/26/04 

Wedl 

2/15/04 

3 

Model  Build  2 

Fri  9/30/05 

Fri 

4 

Testbed  Build  2 

Mon  10/3/05 

Wed ' 

5 

Model  Build  3 

Thu  8/3/06 

Th 

6 

Testbed  Build  3 

Fri  8/4/06 

Thu 

7 

Model  Build  4 

Fri  4/1 3/07 

Fri 

8 

Testbed  Build  4/Final 

Mon  4/1 6/07 

Thu 

9 

10 

SSDS  TE  #1 

Thu  12/8/05 

Sat ' 

i 

11 

SPS-48E  Characterization  Testing 

Mon  12/1 2/05 

Tue ' 

12 

SSDS  TE  # 2 

Mon  4/1 0/06 

Fri 

13 

SESEF/SEMCIP 

Mon  4/1 7/06 

Wed 

14 

SSDS  DTE  TE  #3 

Mon  6/5/06 

Thu  6/8/06 

15 

CSSQT  U/W 

Tue  6/27/06 

Wed  7/1 2/06 

16 

A/C  Trackex 

Mon  6/26/06 

Mon 

17 

DT/DTE 

Tue  6/27/06 

Tue 

18 

MSLEX  (BQM  34) 

Wed  7/5/06 

Th 

19 

RCSR  Measurement 

Thu  7/6/06 

Si 

W 

20 

SSDS  TE  #4 

Mon  10/2/06 

Thu 

vv 

21 

A/C  Trackex 

Mon  10/2/06 

Tue 

To 

22 

DTE 

Wed  1 0/4/06 

Thu 

23 

DT/OT  Firings 

Mon  10/9/06 

Thu ' 

24 

SSDS  DTE  TE  #5 

Mon  1  /S/07 

Thu 

25 

RCSR  Validation 

Mon  4/30/07 

Tu 

26 

SDTS  OP  1 

Tue  7/25/06 

Tue 

27 

SDTS  OP  2 

Tue  8/1 5/06 

Tue 

8/1 5/U6 

28 

SDTS  OP  3 

Thu  8/1 7/06 

Thu  8/1 7/06 

29 

SDTS  OP  4 

Tue  9/1 2/06 

Tue  9/1 2/06 

30 

31 

Dry  Runs 

Mon  9/3/07 

Fri  1 1  /9/07 

32 

Analyis  Runs 

Mon  11  /1 2/07 

Fri  8/1 5/08 

33 

Draft  V&V  Report 

Fri  11/9/07 

Fri  1 1  /9/07 

34 

Preliminary  DT  Accreditation 

Fri  11/9/07 

Fri  1 1  /9/07 

35 

Final  V&V  Report 

Fri  8/22/08 

Fri  8/22/08 

36 

Final  DT  Accreditation 

Fri  9/1 9/08 

Fri  9/1 9/08 

37 

OT  Recommendation  to  COTF 

Fri  10/1 7/08 

Fri  1 0/1 7/08 

Gtr  4 


0  N  D 


2005 


Qtr  1 


J  F  M 


Qtr  2 


A  M  J 


Qtr  3 


J  A  S 


Qtr  4 


0  N  D 


2006 


Qtr  1 


J  F  M 


Qtr  2 


A  M  J 


Qtr  3 


J  AS 


Qtr  4 


0  N  D 


2007 


Qtr  1 


j  If  |m 


Qtr  2 


A  M  J 


Qtr  3 


J  A  S 


Qtr  4 


0  N  D 


2008 


Qtr  1 


J  F  M 


Qtr  2 


A  |  M  |  ,1 


Qtr  3 


J  AS 


Qtr  4 


ON  D 


4  10/2  5 


1600  Runs 
2  Hours  Per  Run 


40  Weeks  for  All  Runs 


And  Overcome  16  Hour  Days 


Analysis  Runs 


4/13 


4  11/9 
4  119 


4  8  22 
4  9-19 


4  1017 


ANALYSIS  -  KEY  EVENTS 
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•  Identify  Key  Events  During  Engagement 

-  Sensor  Performance,  Sensor  Messages,  Weapons 
Orders,  Weapons  Performance,  Engagement 
Outcome 

•  Data  Collected  and  Displayed  Live  During 
Runs 

•  Used  to  Verify,  Troubleshoot  Testbed 
Performance 

•  Used  to  Calculate  Various  PRA  Values 


SUMMARY 
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Success  for  Testbed  Development  Based  On: 

-  Clearly  Defined  Requirements 

-  Testbed  Organization  and  Roles  Well  Understood 

-  Documents  Contain  Necessary  Information 

-  Spiral  Development  and  Schedule 

-  Execute  Phases  of  Simulation  Development 

-  Pragmatic  Scenario  Development 

-  Systematic  Analysis  Approach 

-  Collection,  Manipulation  and  Presention  of  PRA 
Values 


BACKUP  SLIDES 
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A  Case  Study  of  T&E  Data 
Supporting  A  Simulation 


LPD  17  PRA  Testbed 
Vincent  M.  Ortiz 
AVW  Technologies 
9  March,  2006 


OVERVIEW 
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•  LPD  17  San  Antonio  Ship  Class 

•  LPD  17  Probability  of  Raid  Annihilation  (Pra) 
Testbed  Description  and  Architecture 

•  Historical  vs  Integrated  Approach  to  Testing 

•  PRA  Analysis 

•  Validating  the  LPD  17  Testbed 

•  Organizational  Approach  to  Have  T&E  Data 
Support  the  LPD  17  PRA  Testbed 
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LPD  17  CAPABILITIES 


*  The  LPD  17  capabilities  include: 

-  State-of-the-art  command  and  control  suite 

-  Advanced  ship  survivability  features  that  enhance  its  ability 
to  operate  in  the  unforgiving  littoral  environment  (e.g.,  low 
radar  cross  section) 

-  Substantially  increased  landing  force  vehicle  lift  capacity 
(23,600  square  feet  of  vehicle  storage  space) 

-  Large  flight  deck  (land  2  MV-22  or  4  CH-46)  and  well  deck 
(holds  2  Landing  Craft  Air  Cushion  {LCAC}) 

*  The  LPD  17  is  the  first  amphibious  ship  designed  to 
accommodate  the  Marine  Corps’  “mobility  triad” 

-  Expeditionary  Fighting  Vehicle  (EFV) 

-  LCAC 

-  MV-22  Osprey  tilt  rotor  aircraft. 


OUR  FOCUS  WILL  BE  ON  THE  COMBAT  SYSTEM 


Low  Radar  Cross  Section  (RCS) 


SPS-48E  3D  Air  Search  Radar 


SPQ-9B  Air/Surface  Tracking  Radar 


SSDS  Control  and  Decision  System 


Nulka  Active  Decoy 


SLQ  32A(V)2  EW  System 


RAM  Missile  System 


BACKGROUND  -  Pra 
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OBJECTIVE:  ASSESS  LPD  17’s  PRA 
(ABILITY  TO  DEFEND  ITSELF  AGAINST  INCOMING  MISSILES) 

■  CNO’s  Anti-Air  Warfare  Capstone  Requirements  Document  mandated  the  ship 
self  defense  capability  for  specific  ship  classes  and  established  the  PRA  as  the 
primary  Measure  of  Effectiveness  (MOE)  to  assess  ship  combat  system  suites. 

■  PRA  is  defined  as  the  ability  of  a  particular  stand-alone  ship,  as  an  integrated 
system,  to  detect,  control,  engage,  and  defeat  a  specified  raid  of  anti-ship  cruise 
missile  (ASCM)  threats  with  a  specified  level  of  probability  in  the  operational 
environment. 


■  Th®  PRA  MOE  is  a  system -of -systems  measure  which  is  levied  on  the  ship 
defense  suite  as  a  whole  to  properly  detect,  control,  and  engage  (annihilate)  a 
raid  of  incoming  threat  ASCMs.  Thus,  it  doesn’t  measure  the  performance  of  any 
particular  ship  defense  element;  rather  it  measures  the  system  performance  of  all 
the  ship  defense  elements  across  the  complete  battle  timeline. 

■  The  LPD  17  class  is  the  first  U.S.  naval  ship  class  required  to  demonstrate  its 
ability  to  defeat  specific  anti-ship  cruise  missile  threats  to  achieve  a  statistical 
Pra- 
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LPD  17  PRA  TESTBED 


Naval  ResearcfrLab 
Washington,  DC  \ 


Scenario  & 
Environment 
data  drivers 


SIMDIS 


JDARS 


SSDS 


CEP 


RAM 

M-on-N 


RAM 

launcher 


Decoys 


JHU  Applied  Physics  Lab 
Laurel,  MD 


NAWC  Weapons  Division 
China  Lake 


Threats/targets  I  SLq.32  »  SPS-48E  I  SPQ-9B 
seekers  &  airframes  I  w  0! 


Network  Interface  Layer 


g 


Geographically  Distributed  Federation  of  Tactical  HWIL, 
Tactical  SWIL  and  Digital  Physics  Based  Models 


NAVY  CATEGORIES  OF  TESTING 
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•  Land  Based  Test  Site  (LBTS)  Testing 

•  Lead  Ship  Testing/ 

Operational  Testing  (OT) 

-  Each  New  Ship  Class 

-  Each  New  Combat  System  Element 

•  Self  Defense  Test  Ship  (SDTS)  and  Test 
Events 

•  Pra  Modeling  and  Simulation 

Navy  Initiative  Underway  to  Combine  and  Optimize  Testing  of  New  Systems 
To  Eliminate  Duplicate  Efforts  and  To  Achieve  Cost  Savings 


NAVY  INTEGRATED  TESTING 
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*  Integrate  Planning,  Resourcing,  Budgeting 
and  Execution  Across  Combat  System 
Variants  and  Associated  Elements 

-  No  Longer  Planned  Independently  by  Each 
Program  Office 

*  Optimization  Efforts  Include: 

-  Maximize  Combat  System  Ship  Qualification  Test 
(CSSQT)  Resulting  in  Less  DT,  OT 

-  Leverage  Other  Ship  Class  Combat  System 
Testing 

-  Testing  of  Common  Variant 

-  Maximize  SDTS  Testing  Events 

-  Maximize  Use  of  M&S  (Pra  &  Other  Simulations) 
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LPD  17  SOLUTION  TO  Pra 


/  /  \ 


Pra  Assessment  is  a 
Three  Pronged  Approach 

-  Test  Against  Actual  Ship  (LPD  17) 

•  Pro  -  Test  Targets  Against  the  Actual  Ship 

•  Con  -  Limited  Firing  Events, 

Cannot  Fire  Target  Directly  at  Ship 


-  Test  Against  SDTS 

•  Pro  -  Targets  and  Actual  Threats,  Profile  is  Closer  to  SDTS 

•  Con  -  Limited  Representation  of  the  Actual  Ship, 

Limited  Firing  Events 

-  Test  Using  M&S  (LPD  17  Pra  Testbed) 

•  Pro  -  Can  Runs  Numerous  Threats,  Scenarios,  Events 

•  Con  -  Developmental  Cost  &  Time,  Limiting  Assumptions 
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OPTIMIZED  TESTING  -  LPD  17 


•  CSSQT 

-  Combat  System  Ship  Qualification  Testing  (Prove  Out  the  CS) 

-  Maximize  Use  of  Detect  to  Engage  Sequence  to  Satisfy  DTI  OT 
Requirements 

-  Help  Resolve  Pra  Measure  of  Effectiveness  (MOE) 

•  Lead  Ship/  Operational  Testing 

-  Tracking  Exercises 

-  Target  Firings,  Combat  System  Detect  to  Engage  Sequence 

-  Nulka  Testing 

-  Help  Resolve  Pra  MOE 

•  SDTS 

-  Target  Firings,  Engagement  Analysis  of  Stressing  Targets 

-  Help  Resolve  PRA  MOE 

•  Pra  Testbed 

-  Data  Collection  from  Above  Firings  for  Validation 

-  Pra  MOE  Analysis  (Testbed  Accredited  Specifically  for  Pra) 

-  Feedback  of  Combat  System  Performance  to  Developers 

-  Not  Used  for  Preflight  Predictions  for  Target  Firings 


OPTIMIZING  T&E  AND  M&S 
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*  The  ‘Chicken  and  the  Egg’  Dilemma 

-  You  Need  the  Data  to  Accredit  the  Testbed 

to  Perform  the  Preflight  Predictions  for  the  Live  Fire  Events 
that  Get  the  Data 

*  M&S  Optimizes  Its  Use  of  T&E  Data 

-  Use  Tracking  and  Live  Fire  Data  for  Validation 

-  Integrate  Validation  Results  Into  the  Testbed 

-  Validate  and  Accredit  the  Testbed 

*  T&E  Data  Optimizes  Its  Use  of  M&S 

-  Live  Fire  Events  Use  Stand  Alone  Models  For  Preflight 
Predictions 

-  Testbed  Runs  Gain  Understanding  of  Combat  Systems 
Sensitivities  (Not  Accredited  to  Perform  Preflight 
Predictions) 

-  In  the  Future  -  Accredit  the  Testbed  to  Perform  Preflight 
Predictions  (Although  it  Needs  Live  Fire  Data  to  Accredit?) 


LPD  17  Pra  TESTBED 


*  Spiral  Development 

-  4  Builds  Over  4  Years 

*  Validation  Activities 

-  Compare  Event  With  Replicated  Event  In  the  Testbed 

•  CSSQT  Event 

•  Lead  Ship/  OT  Firings,  Tracking  Exercises 

-  Integrated  Validation  Data  Into  Testbed 

*  Analysis  Approach 

-  20  Pra  Events  (5  Targets,  2  Geographic  Locations,  2  Ship 
Signatures) 

-  80  Runs  Per  Pra  Event  (5  Times  of  Day,  8  Threat  Radials,  2 
Seasons) 

-  One  Firing  for  Each  Unique  Run 
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Threat  1 
T  2 
T  3 
T  4 
T  5 


TESTBED  SAMPLE  SPACE 


PRA  Event 

Med  SOH 

Geographic  Location 


PRA  Event  =  20 

(5  Threat  x  2  GEO  x  2  Signature) 
PRA  Run  =  80 

(5  Time  x  8  Radials  x  2  Seasons) 
One  Firing  for  Each  Unique  Run 
Total  Number  of  Runs  =  1600 


Clean  RCS  Ship 
Dirty  RCS  Signature 


PRA  Run 


Time: 

Sunrise 

Noon 

Afternoon 

Sunset 

Midnight 


8  Threat 
Radials 

PRA  Event 


Season: 
//  Winter 
Summer 
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TESTBED  SCHEDULE 
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LPD  17  PRA  ORGANIZATION 


MANAGEMENT  IPT 

*  LPD  17  Combat  System  Integration  Manager  *  Ship  Self  Defense  Combat  Systems  Engineer 

•  LPD  17  Test  Director  •  Deputy  SSD  CSE 


WORKING  IPT 


CS  Element  PMs 


SIMULATION  CONTROL  PANELS  (SCP) 


Threat 

! 

Softkill 

■ 

Radars 

Natural 

Environment 

1 _ 

Hardkill 

1 _ 

CEC 

■ _ 

M&S  Developers 

4 


Scenario 


Test  Bed 


SSDS 
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ROLES  &  RESPONSIBILITIES 


PMS317 

■  Manage  Funding 

■  Drive  Schedule 

■  V&V  Manager 

■  DT  Accrediting  Authority 

PEO  IWS  CSE 

■  Manage  Testbed  Design 
and  Development 

NRL 

■  Testbed  Integrator 

NSWC  Corona 

■  Test  Resource,  Planning 
and  Data  Collection  Agent 


Element  PMs 

■  Co-Chair  SCP 

■  Review  &  Approve  SOWs 
associated  with  M&S 
Development 

■  Manage/  Participate  in 
Model  Development 

■  Responsible  for  the 
Credibility  of  their 
Respective  Models 

Model  Developers 

■  Develop/  Integrate  Models 

COMOPTEVFOR 

■  Participates  as  the  OT 
Accrediting  Authority 


COLLECTING  VALIDATION  DATA 
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•  LPD  17  Organization 

-  Dedicated  Test  Planning  Position 

*  Experienced  Tester  -  Understands  The  Community 

*  Knowledgeable  in  LPD  17  Testbed  Process 

•  Data  Collection  Process 

-  Supports  the  Generation  of  the  Live  Testing  Data 
Needs 

•  Determine  What  Testbed  Developers  Need 

•  Put  Needs  into  a  Document  that  Live  Testers  Understand 

-  Involved  in  the  Actual  Tests 

Close  Working  Relationship  with  Live  Testers 
Vital  in  Collection  of  Needed  Live  Data 


DATA  COLLECTION  PRINCIPLES 
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•  Early  Involvement 

•  Establish  A  Strong  Working  Relationship 
Between  Developers  and  Testers 

•  Clearly  Define  Data  Collection  Needs 

-  Understand  What  Developers  Want 

-  Articulate  Into  What  Testers  can  Understand, 
Collect 

•  Effective  Communication 

-  Meetings 

-  Working  Documents 

•  Arrive  at  a  Finalized  Set  of  Events  and  Data 
Collection  that  will  Support  the  Testbed 
Validation 
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A  Case  Study  of  T&E  Data 
Supporting  A  Simulation 


Questions? 


BACKUP  SLIDES 
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AVW  Process  developed  the  Approach,  Requirements  and  Build  Plan 
AVW  Database  Produced  the  Requirements  and  VV&A  Documents 
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LPD  17  Pra  TESTBED  OVERVIEW 


MANAGEMENT  APPROACH: 


Organization 

Meetings 

Documents 

Schedule 


BOUND  THE  PROBLEM: 

Testbed  Requirements 
Fidelity 

Ship  Configuration 
Environment 
Threat  Types 
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TECHNICAL  APPROACH: 


Physics  -  Based 
Non  -  Real  Time 
Distributed,  RTI  Solution 
HLA  Compliant 
Spiral  Development 


BOUND  THE  ANALYSIS: 

Finite  Number  of  Runs 
f(Geographic  Location 
Ship  Configuration 
Season,  Time  of  Day 
Threat  Types) 


AVW  TESTBED  SPIRAL  DEVELOPMENT 


TECHNOLOGIES 


n 

i 

x 

Develop  Model 
Build/  CM  Plan  and 
V&V  Plan 


Develop  the  Models  and  Testbed 
Through  Spiral  Development 
Of  Four  Successive  Builds 


Submit: 
Final  Report, 
Accreditation 
Package 


DEVELOPMENT  TIMELINE 


Build 

1 
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Time 
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SCENARIO 


Banghazr 


jtta 


Geography  1 
Open  Ocean  -  Mid-Med 


GEOGRAPHIES 


Geography  2 
Straits  of  Hormuz 


ANALYSIS  APPROACH 
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•  2  Geographies 

-  Med  Open  Ocean 

-  Straits  of  Hormuz 

-  Provides  Stressing  and 
Non-Stressing  Locations 

•  2  Environments 

-  2  Times  of  Year 

-  5  Times  of  Day 

-  No  Rain 

-  Provides  Nominal 
Changes  in  Environment 


2  Radar  Cross  Sections 

-  Clean,  Minimized  RCS 

-  Dirty,  open  well,  helo  on  deck 

-  Provides  Large  and  Small 
Signatures 

5  Threats 

-  T1R1,  T2,  T3,  T5,  T7 

-  8  Threat  Bearings 

-  45  Deg  Intervals 

-  Provides  Combat  System 
Performance  from  all 
Directions 


PERFORM  ONE  RUN  FOR  EACH  COMBINATION  OF  6  VARIABLES 
STATISTICALLY  A  REPRENTATIVE  SAMPLING  THROUGH  THE  SPACE 
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Threat  1 
T  2 
T  3 
T  4 
T  5 


TESTBED  PRA  CALCULATIONS 


PRA  (Event)  = 
PRA  (Threat)  = 


PRA  Event 

Med  SOH 

Geographic  Location 


#  Successes  (20  PRA  Values) 
80 

I  PRA  Events  (5  PRA  Values) 
4 

PRA  (Geography)  =  I  PRA  Events  (2  PRA  Values) 

10 

PRA  (Ship  Sig)  =  I  PRA  Events  (2  PRA  Values) 

10 

PRA  Overall  =  I  PRA  All  Event  (1  PRA  Value) 


PRA  Event  =  20 

(5  Threat  x  2  GEO  x  2  Signature) 
PRA  Run  =  80 

(5  Time  x  8  Radials  x  2  Seasons) 
One  Firing  for  Each  Unique  Run 
Total  Number  of  Runs  =  1600 


Clean  RCS  Ship 
Dirty  RCS  Signature 


PRA  Run 


Time: 

Sunrise 

Noon 

Afternoon 

Sunset 

Midnight 


8  Threat 
Radials 

PRA  Event 


Season: 
//  Winter 
Summer 
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TESTBED  SCHEDULE 


ID 

Task  Name 

Start 

Finish 

1 

Model  Build  1 

Mon  10/25/04 

Mon  1 0/25/04 

2 

Testbed  Build  1 

Tue  10/26/04 

Wedl 

2/15/04 

3 

Model  Build  2 

Fri  9/30/05 

Fri 

4 

Testbed  Build  2 

Mon  10/3/05 

Wed ' 

5 

Model  Build  3 

Thu  8/3/06 

Th 

6 

Testbed  Build  3 

Fri  8/4/06 

Thu 

7 

Model  Build  4 

Fri  4/1 3/07 

Fri 

8 

Testbed  Build  4/Final 

Mon  4/1 6/07 

Thu 

9 

10 

SSDS  TE  #1 

Thu  12/8/05 

Sat ' 

i 

11 

SPS-48E  Characterization  Testing 

Mon  12/1 2/05 

Tue ' 

12 

SSDS  TE  # 2 

Mon  4/1 0/06 

Fri 

13 

SESEF/SEMCIP 

Mon  4/1 7/06 

Wed 

14 

SSDS  DTE  TE  #3 

Mon  6/5/06 

Thu  6/8/06 

15 

CSSQT  U/W 

Tue  6/27/06 

Wed  7/1 2/06 

16 

A/C  Trackex 

Mon  6/26/06 

Mon 

17 

DT/DTE 

Tue  6/27/06 

Tue 

18 

MSLEX  (BQM  34) 

Wed  7/5/06 

Th 

19 

RCSR  Measurement 

Thu  7/6/06 

Si 

W 

20 

SSDS  TE  #4 

Mon  10/2/06 

Thu 

vv 

21 

A/C  Trackex 

Mon  10/2/06 

Tue 

To 

22 

DTE 

Wed  1 0/4/06 

Thu 

23 

DT/OT  Firings 

Mon  10/9/06 

Thu ' 

24 

SSDS  DTE  TE  #5 

Mon  1  /S/07 

Thu 

25 

RCSR  Validation 

Mon  4/30/07 

Tu 

26 

SDTS  OP  1 

Tue  7/25/06 

Tue 

27 

SDTS  OP  2 

Tue  8/1 5/06 

Tue 

8/1 5/U6 

28 

SDTS  OP  3 

Thu  8/1 7/06 

Thu  8/1 7/06 

29 

SDTS  OP  4 

Tue  9/1 2/06 

Tue  9/1 2/06 

30 

31 

Dry  Runs 

Mon  9/3/07 

Fri  1 1  /9/07 

32 

Analyis  Runs 

Mon  11  /1 2/07 

Fri  8/1 5/08 

33 

Draft  V&V  Report 

Fri  11/9/07 

Fri  1 1  /9/07 

34 

Preliminary  DT  Accreditation 

Fri  11/9/07 

Fri  1 1  /9/07 

35 

Final  V&V  Report 

Fri  8/22/08 

Fri  8/22/08 

36 

Final  DT  Accreditation 

Fri  9/1 9/08 

Fri  9/1 9/08 

37 

OT  Recommendation  to  COTF 

Fri  10/1 7/08 

Fri  1 0/1 7/08 

Gtr  4 


0  N  D 


2005 


Qtr  1 


J  F  M 


Qtr  2 


A  M  J 


Qtr  3 


J  A  S 


Qtr  4 


0  N  D 


2006 


Qtr  1 


J  F  M 


Qtr  2 


A  M  J 


Qtr  3 


J  AS 


Qtr  4 


0  N  D 


2007 


Qtr  1 


j  If  |m 


Qtr  2 


A  M  J 


Qtr  3 


J  A  S 


Qtr  4 


0  N  D 


2008 


Qtr  1 


J  F  M 


Qtr  2 


A  |  M  |  ,1 


Qtr  3 


J  AS 


Qtr  4 


ON  D 


4  10/2  5 


1600  Runs 
2  Hours  Per  Run 


40  Weeks  for  All  Runs 


And  Overcome  16  Hour  Days 


Analysis  Runs 


4/13 


4  11/9 
4  119 


4  8  22 
4  9-19 


4  1017 
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LPD  17  PRA  Testbed  W&A  Database: 
A  Disciplined  Approach  for  W&A 


Vincent  M.  Ortiz 
AVW  Technologies 
9  March,  2006 
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W&A  DATABASE  OVERVIEW 


•  An  Example  of  Making  W&A  Work 

•  The  Simulation 

•  The  Simulation  Development  Process 

•  The  W&A  Approach 

•  The  W&A  Process 

•  The  W&A  Database 


W&A  DATABASE  OVERVIEW 
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•  An  Example  of  Making  W&A  Work 

-  Have  Completed  Build  2  of  the  4  Build  LPD  17 
Probability  of  Raid  Annihilation  (Pra)  Testbed 

-  Have  Successfully  Integrated  the  W&A  Process 
into  the  Development  Cycle 

-  The  Documentation  is  Tracked  via  a  Relational 
Database 

•  Describe  the  Simulation 

•  Describe  the  Simulation  Process 

•  Describe  the  W&A  Approach 

•  Describe  the  W&A  Process 

•  Describe  the  W&A  Database 
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W&A  DATABASE  OVERVIEW 


•  An  Example  of  Making  VV&A  Work 

•  The  Simulation 


-  LPD  17  San  Antonio  Ship  Class 

-  LPD  17  Combat  System 

-  Pra  Requirement  Definition 

-  LPD  17  Pra  Testbed  Simulation 


•  The  Simulation  Process 

-  Management,  Technical  Approach,  Bound 
Problem  Space,  Defined  Analysis  Approach 

•  The  VV&A  Approach 

•  The  VV&A  Process 

•  The  VV&A  Database 

•  Relational  Database  Tables 
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LPD  17  CAPABILITIES 


*  The  LPD  17  capabilities  include: 

-  State-of-the-art  command  and  control  suite 

-  Advanced  ship  survivability  features  that  enhance  its  ability 
to  operate  in  the  unforgiving  littoral  environment  (low  radar 
cross  section) 

-  Substantially  increased  landing  force  vehicle  lift  capacity 
(23,600  square  feet  of  vehicle  storage  space), 

-  Large  flight  deck  (land  2  MV-22  or  4  CH-46)  and  well  deck 
(holds  2  Landing  Craft  Air  Cushion  {LCAC}) 

*  The  LPD  17  is  the  first  amphibious  ship  designed  to 
accommodate  the  Marine  Corps’  “mobility  triad” 

-  Expeditionary  Fighting  Vehicle  (EFV) 

-  LCAC 

-  MV-22  Osprey  tilt  rotor  aircraft. 


OUR  FOCUS  WILL  BE  ON  THE  COMBAT  SYSTEM 


Low  Radar  Cross  Section  (RCS) 


SPS-48E  3D  Air  Search  Radar 


SPQ-9B  Air/Surface  Tracking  Radar 


SSDS  Control  and  Decision  System 


Nulka  Active  Decoy 


SLQ  32A(V)2  EW  System 


RAM  Missile  System 


BACKGROUND  -  Pra 
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OBJECTIVE:  ASSESS  LPD  17’s  PRA 
(ABILITY  TO  DEFEND  ITSELF 
AGAINST  INCOMING  MISSILES) 

■  CNO’s  Anti-Air  Warfare  Capstone  Requirements  Document 
mandated  the  ship  self  defense  capability  for  specific  ship 
classes  and  established  the  P,^  as  the  primary  Measure  of 
Effectiveness  (MOE)  to  assess  ship  combat  system  suites. 

■  PRA  is  defined  as  the  ability  of  a  particular  stand-alone  ship,  as  an 
integrated  system,  to  detect,  control,  engage,  and  defeat  a 
specified  raid  of  anti-ship  cruise  missile  (ASCM)  threats  with  a 
specified  level  of  probability  in  the  operational  environment. 

■  The  LPD  17  class  is  the  first  U.S.  naval  ship  class  required  to 
demonstrate  its  ability  to  defeat  specific  ASCM  threats  to  achieve 
a  specified  P,^. 
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LPD  17  PRA  TESTBED 


Naval  Research  Lab 
Washington,  DC  \ 


Threats/targets 
seekers  &  airframes 


SPS-48E 


SPQ-9B 


JDARS 


Scenario  & 
Environment 
data  drivers 


SSDS 


CEP 


RAM 
M-on-N 


JHU  Applied  Physics  Lab 
Laurel,  MD 


Network  Interface  Layer 


RAM 

launcher 


SIMDIS 


Decoys 


NAWC  Weapons  Division 
China  Lake 


g 


Geographically  Distributed  Federation  of  Tactical  HWIL, 
Tactical  SWIL  and  Digital  Physics  Based  Models 
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LPD  17  Pra  TESTBED  OVERVIEW 


MANAGEMENT  APPROACH: 


Organization 

Meetings 

Documents 

Schedule 


BOUND  THE  PROBLEM: 

Testbed  Requirements 
Fidelity 

Ship  Configuration 
Environment 
Threat  Types 
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TECHNICAL  APPROACH: 

Physics  -  Based 
Non  -  Real  Time 
Distributed,  RTI  Solution 
HLA  Compliant 
Spiral  Development 


BOUND  THE  ANALYSIS: 


Finite  Number  of  Runs 
(Geographic  Location 
Ship  Configuration 
Season,  Time  of  Day 
Threat  Types) 


W&A  DATABASE  OVERVIEW 

•  An  Example  of  Making  VV& A  Work 

•  The  Simulation 

•  The  Simulation  Process 

•  The  W&A  Approach 

-  Set  up  Process  with  Defined  V&V  Checks 

-  Leverage  off  of  Previous  Accreditation  Packages, 
Focus  on  Implementation  in  the  Testbed 
Simulation 

-  Integrate  V&V  into  the  Simulation  Spiral 
Development 

-  Have  a  Dedicated  V&V  Team  to  Relieve  Pressure 
from  Developers 

•  The  W&A  Process 

•  The  W&A  Database 
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4  PHASES  of  V&V  AND  CHECKS 


*  As  Defined  in  the  DOD  M&S  Recommended 
Practices  Guide  (RPG) 

*  1.  Conceptual  Model  (and  SECM)  Validation 

-  Conceptual  Model  -  Testbed  Design  and  Architecture 

-  SECM  -  System  Engineering  Conceptual  Model  Document 

*  2.,  3.  Functional  Design  and  System  Verification 

-  Combine  Functional  Design  Verification  Step  with  The 
System  Verification  Step 

-  Verify  Data  within  the  Models 

*  4.  Results  Validation 

-  Use  Live  Test  Data  to  Validate  Testbed  Performance 

*  Data  Verification 

-  Defined  as  Environmental,  Scenario,  and  FOM  Data 
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LPD  17  Pra  M&S  AND 
W&A  PROCESSES 


4  PHASES  of  V&V  -  STEP  1 


•  Conceptual  Model  and  SECM  Validation 

-  Conceptual  Model  Validation 

•  Review  Individual  Models  Ability  to  Satisfy  Requirements 

•  Review  Model’s  Role,  Interactions  Within  the  Testbed 

-  System  Engineer  Conceptual  Model  Validation 

•  Review  Universal  Modeling  Language  Representation 

•  Review  Input,  Output  Flows  For  Each  Model 

-  Trace  Requirements  to  Models,  Model  Elements 
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4  PHASES  of  V&V  -  STEP  2/3 


*  Functional  Design  and  System  Verification 
(Partial  Listing) 

-  HW  Architectural  Design  Review 

-  Design  Review  of  Rehosted  Tactical  Code 

-  Algorithm  and  Structure  Control  Flow 

-  Evaluate  Interfaces 

-  Model  Input/Output  Visualization 

-  Model  Element  Black  Box  Functionality 

-  SME  Model  to  Testbed  Input/Output  Comparison 

-  Verify  Input  Data /  Output  Data  as  Appropriate 

-  Trace  Requirements  into  Design 

-  Model  Performance  Compliance 


4  PHASES  of  V&V  -  STEP  4 


*  Results  Validation 

-  Display  Model  Execution 

-  Model  Output  Data  -  Format  and  Fidelity 

-  Operationally  Test  Model  for  Proper  Operation 

-  SME  Comparison  of  Model  to  Actual  System 

-  Test  Federation  Requirements 

-  Validate  Model  Output  Using  Real-World  Input 
Data 

-  Trace  Requirements  to  Model  Performance 

-  Model  Performance  Compliance 
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4  PHASES  of  V&V 


*  Data  Verification 

-  Assess  Environmental  Data 

•  Verify  Transformation/  Data  Consistency 

•  Verify/  Validate  Data  and  Metadata 

•  Verify/  Validate  Initialization  Data 

-  Assess  Scenario  Data 

•  Verify  Transformation/  Data  Consistency 

•  Verify/  Validate  Scenario  Data  Set 

•  Verify/  Validate  Data  and  Metadata 

-  Assess  FOM  Data 

•  Graphical  Comparison 

•  Verify  Object  Attributes  and  Structure 

•  Verify  Interaction  Parameters  and  Data  Types 
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LEVERAGE  PREVIOUS  W&A 
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•  Review  Model’s  Previous  Accreditation 
Package 

-  For  Model  Credibility 

-  For  Applicability  to  Testbed 

•  W&A  Team  Focus 

-  The  Model  as  it  is  Used  Within  the  Testbed 

-  Integration  of  the  Model  Within  the 
Testbed 

-  Model  Interfaces  Within  the  Testbed 
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V&V  AND  MODEL  DEVELOPMENT 


•  LPD  17  PRA  Testbed  Spiral 
Development  Approach 

-  Four  Builds  Over  3  Years 

-  Increasing  Functionality  Within  Each  Build 

•  V&V  Integration 

-  Identify  V&V  Checks  That  Can  Be 
Performed  During  the  Builds 

-  Perform  Checks  at  the  Completion  of  Each 
Build 


DEVELOPMENT  TIMELINE 


Build 

1 
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Time 
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LPD  17  PRA  ORGANIZATION 


MANAGEMENT  IPT 

*  LPD  17  Combat  System  Integration  Manager  *  Ship  Self  Defense  Combat  Systems  Engineer 

•  LPD  17  Test  Director  •  Deputy  SSD  CSE 


WORKING  IPT 


CS  Element  PMs 


SIMULATION  CONTROL  PANELS  (SCP) 


Threat 

Softkill 

I 

■ 

Radars 

Natural 

Environment 

1 _ 

m 

Hardkill 

1 _ 

1 

CEC 

M&S  Developers 

4 


Scenario 


Test  Bed 


SSDS 
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W&A  ORGANIZATION 


•  Separate  Team  From  Developers 

-  Experienced  in  Combat  Systems  and  Ship 
Operations 

-  Knowledgeable  in  Verification  and  Validation 
Process 

•  V&V  Philosophy 

-  V&V  Team  Perform  the  V&V  Checks  (with 
Assistance  of  the  Developers  as  Necessary) 

-  V&V  Checks  Performed  During  Each  Build  as  the 
Testbed  Functionality  Permits 

-  V&V  Team  Generates  the  Documentation 

-  Minimizes  the  Workload  on  the  Developers 
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W&A  DATABASE  OVERVIEW 


•  An  Example  of  Making  VV&A  Work 

•  The  Simulation 

•  The  Simulation  Process 

•  The  VV&A  Approach 

•  The  W&A  Process 


-  Requirements  is  the  Foundation, 

(there  are  over  1600  for  this  Federation) 

-  Arrange  Requirements  under  Models,  Builds 

-  Assign  V&V  Checks,  Acceptability  Criteria  to  Each 
Requirement 

-  Perform  V&V  During  Each  Build 

-  Generate  V&V  Reports 

•  Describe  the  VV&A  Database 
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W&A  DATABASE  OVERVIEW 


•  An  Example  of  Making  W&A  Work 

•  The  Simulation 

•  The  Simulation  Process 

•  The  W&A  Approach 

•  The  W&A  Process 

•  The  W&A  Database 


-  Relational  Database  Tables 

-  User  Interface 

-  Database  Reports  That  can  be  Generated 

-  Program  is  Easily  Modified,  Updated. 

-  Data,  Reports  is  Property  of  the  Customer 
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LPD  17  Pra  W&A  DATABASE 


*  Maps  Requirements  to  Testbed/ 

Models/  Model  Elements 

*  Maps  Requirements  to  Builds 

*  Maps  V&V  Activities  to  Requirements/  Testbed/ 
Elements/  Subelements/  Builds 

*  Tracks  Completion  of  V&V  Activities 

*  Includes  Comments/  Results/  V&V  Documents 

*  The  Database  is  Capable  of  Printing  a  Variety  of 
Documents  for  W&A  Reports,  etc. 

*  Uses  Live  Test  Events  for  Validation 
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AVW  W&A  DATABASE 


•  Microsoft  Access/VBA  Relational  Database 

-  User  Friendly,  Uncomplicated  and  Customizable 

-  Low  costs  in  License  and  Tech  Support 

-  NMCI  Compatible 

•  Supports  process  standardization 

-  Consistent  with  M&S  Instructions 

-  Buy  in  from  COTF,  DOT&E 

•  Inherent  flexibility  of  a  database 

-  Reports  standardization 

-  Query  for  specific  or  tailored  reports 

•  Assists  COTF  and  PM 

-  Provides  quick,  easy  access  to  all  information  requested 

-  Provides  single  source  for  requirements  traceability  to  all  W&A 
efforts 

-  Manages  associations  from  requirements  to  development  to  VV&A 
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DECOY  MODEL  BUILD  2  REPORT 
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•  A  Disciplined  Approach 

-  Defined  Management,  Technical  Framework 

-  Consistent  with  M&S  Guidance 

•  A  Developed,  Working  Database 

-  Little  Cost  to  Adapt  to  a  New  Program 

•  Experienced  Personnel 

-  Understand  the  Process  and  the  Potential  Pitfalls 

•  Process  Proven  on  a  Complex  Program 

-  The  Database  and  System  Guides  the 
Development  and  the  V&V  of  the  Simulation 

-  An  Accepted  Process  by  COTF  (Accreditation 
Authority)  and  DOT&E 
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Introduction 


•  CBR  Building  Protection  Overview 

•  Role  of  M&S  and  T&E  in  the  building  protection 
process 

•  Types  of  models  and  experimentation 

•  Interactions  between  nodal  modeling  and 
contaminant  transport  experiments 

•  Example  of  modeling  and  experimental 
interactions  and  conformance  analysis  process 

•  Summary 


CBR  Building  Protection  Overview 


Why  are  buildings  vulnerable  to  CB  attack? 

•  Containment  of  CB  agents  within  a  confined  space  allows 
concentrations  to  rapidly  reach  and  sustain  lethal  levels 


Population  densities  are  high 
in  buildings 

Potential  to  deliver  agent  covertly 

Numerous  adsorbing  surfaces 
that  make  building  restoration 
difficult 


CB  agents  are  effectively  transported  throughout  a  building 
by  mechanical  systems 


COST 


Range  of  Protection  Solutions 


Millions — 


Thousands — 


Hundreds — 


i 

Low 


i 

Medium 

LEVEL  OF  PROTECTION 


Protection  System  Development 


Threat  Assessment 

T 

vulnerability  Assessment 

- - 

Protection  Assessment 

- ► 

i 

System  Design 

◄ - 

T 

installation  and  1  cstmg 

— — 

Training 

i 

Operation  and  Maintenance 
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External  Threat  Modeling 

Baseline  Transport  Modeling 
Characterization  Testing 

Potential  Solution  Development 

Modeling  Analysis  of  Alternatives 

Modeling  Inputs  to  Test  Plan 
Model  Updates  from  Test  Data 
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Types  of  Models 


Ambient  Dispersion  Modeling 

Used  to  characterize  external  threats 
Tools  include  HPAC,  VLSIrack.  Aloha  ... 
Difficult  to  valii 


Nodal  Modeling 

Used  to  characterize  internal  transport  and 
evaluate  protection  system  performance 
Tools  include  CONTAMW  and  COMIS 
Allows  conducting  numerous  model  runs  quickly 
Validate  using  dosage  measurements 
throughout  building 


CFD  Modeling 

Used  to  Integrate  outdoor  and  indoor  models  and  to 
characterize  flow  dynamics  within  rooms 
Time-consuming  to  configure  and  run 
Validation  requires  distributed  concentration  vs.  time 
measurements 
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CONTAMW  Nodal  Modeling 


•  Designed  for  characterization  of  contaminant  transport  though 
ventilated  buildings 

-  Utilities  to  simulate  building  HVAC  systems  and  components 

-  Libraries  with  representative  building  leakage  data 

-  Model  output  of  zone  concentration  profiles  and  flow-path  airflows 

-  GUI  for  simple  model  construction. 


Limitations . . . 

. . .  Solutions 

•  Well-mixed  assumption  inaccurate  for 
larger  building  volumes. 

•  Break  large  volumes  into  subzones  and/or 
correlate  test  data  (parameterizations)  with 
model. 

•  Inaccurate  contaminant  transport  time 
scales. 

•  Correlate  test  data  with  model  and/or  apply 
CFD  modeling  to  large  volumes. 

•  Cannot  model  external  releases. 

•  Characterize  external  cloud  using  ambient 
dispersion  models. 

•  Utilize  CFD  or  parameterizations  to  correct 
for  plume/building  wake  interaction. 

•  CB  agent  properties  not  fully  represented. 

•  Post  process  model  results  with  corrections 
derived  from  test  data  for  deposition  rates, 
release  efficiencies,  removal  mechanisms, 
etc.  | 

Modeling  and  Experimentation 
Requirements 


Nodal  Modeling 

-  Knowledge  of  threat  agent  characteristics 

-  Knowledge  of  building  environment 

-  Understanding  of  limitations  and  solutions  to  limitations 

-  Automated  post-processing 

-  Experience  in  interpreting  model  results 

-  Methods  for  modeling  personnel  movement 


Experimentation 

-  Simulant  to  agent  correlations 

-  Controllable  release  mechanisms  for  repeatable  releases 

-  Sampling  instrumentation,  sample  handling  and  analysis 
methods 

-  Data  analysis  methods 
(including  uncertainty  analysis) 
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Modeling  /  Experimentation  Process 


Modeling  /  Experimentation  Process 
Example 


Example  Building 

•  Former  military  barracks,  30,000  ft2 

•  Three  stories  with  a  quarter  basement 

•  Four  HVAC  zones 


CONTAM  Model  Schematic 


Test  Parameter  Selection  -  Sampling 
Locations 


•  Selection  of  sampling  locations 

-  Release  room  and  adjacent  rooms 

-  HVAC  system  returns,  supplies  and  fresh  air  intakes 

-  At  primary  transport  pathways 

-  In  sets  of  representative  rooms 


- B202  - ELEV  B202A  B226G  - B219  - B201  - B225F  B226F  B220 


Test  Parameter  Selection  -  Release  Mass 


Determination  of  mass  of 
simulant  to  be  released 

-  Release  mass  chosen  to  achieve 
detection  but  not  saturate  real-time 
detectors  (release  room  may  be  an 
exception) 

-  Release  mass  chosen  to  maximize 
measurable  dosages  throughout 
building. 


Real-time 

Detector 

Location 

Max  Release 
Mass  to  Saturate 
Detector,  g 

Min  Release 
Mass  to  Achieve 
Detection,  g 

Release  Room 

3 

0.01 

Room  2 

85 

0.21 

Room  3 

66 

0.17 

Room  4 

64 

0.16 

Test  Parameter  Selection  -  Sampling  Time 


Determination  of  sampling  time 

-  Duration  of  experiment  set  so  that  additional  sampling  time  will  not 
significantly  affect  measured  dosages 


— ♦ — 30  minute  sample  time  — ■ — 1  hr  sampling  time  —A— 2  hr  sampling  time  — x—  4  hr  sampling  time 


Modeling  &  Experimentation  Conformance 
Analysis  Process 


Dosage 


Modeling  &  Experimentation  Conformance 


Conformance  Analysis  Example 
Model  to  Test  Data  Comparison 


1000  . 


0.1  J 

Location 


A  Test  Dosages 

—  Uncertainty 

—  Uncertainty 

- ♦ - Modeled  Dosages 

. Instrument  Method  Detection  Limit 

•Comparison  of  data  shows  deviations  between  model  and  test  data 


Modeling  &  Experimentation  Conformance 


Conformance  Analysis  Example 
Correlated  Model  to  Test  Data  Comparison 

1000  , 


0.1  J - 

Location 


A  Test  Dosages  —  Uncertainty 

—  Uncertainty  . Instrument  Method  Detection  Limit 


« - Modeled  Dosages 


•  Fresh  air  flow-rate  model  adjustment  brings  model  into  better 
agreement  with  experimental  data. 

•  Subsequent  analysis  of  all  experiments  in  set  indicate  adjustment 
improves  or  maintains  conformance. 


Conclusions  and  Lessons  Learned 


•  Integration  of  modeling  and  experimentation  efforts  is 
necessary  to  deal  with  the  shortcomings  of  each. 

-  Instantaneous,  well-mixed  assumption  of  nodal  models 

-  High  cost  of  experimentation 

•  Using  modeling  to  support  planning  of  experiments 
improves  efficiency  in  conduct  of  experiments. 

•  Conformance  analysis  provides  an  effective  means  of 
comparing  modeled  data  to  experimental  data  and 
identifying  model  improvements  to  enhance  fidelity  of 
model  predictions. 

-  Conformance  analysis  must  be  applied  to  all  components  and  all 
test  cases. 
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Open  Technology 
Development 
&  Testing 
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Network-Centric  Systems 

Can’t  create  new  systems  with  old 
tools/processes 

Current  methods  of  acquisition  are  good 
when  purchasing  static  componentry 
Not  so  good  at  acquiring  systems  which  need 
to  be  modular,  networked,  dynamic,  open  to 
unknowable  future  concepts  of  operation. 

Fostering/enabling  innovation  is  central  to 

network-centric  warfare 


INCREASE 

TRANSACTION  RATES* 


•  " The  future  is  here.  It’s  just  not  evenly 
distributed.  ”  -  William  Gibson 


•  “If  you  want  to  succeed,  double  your  failure 
rate”  -  Thomas  Watson,  Founder  IBM 


*  Col.  John  Boyd 


Problem  1 :  Current 
Acquisitions  System 


Requirements  and  acquisitions  process  takes 
too  long 

Needs  in  the  field  aren’t  being  addressed  in 
time  to  have  impact 

Cost  estimates  for  major  weapons  systems 
continually  increasing 

Systems  tend  to  be  used  to  get-the-job  done 
versus  by-the-book 


Problem  Kapidly  Changing 

Threats 

Opponents  able  to  plan  around  our  current  and 
future  planned  strengths  and  capabilities 

Implications: 

Capabilities  built  to  meet  a  moving  target 
->Red  Queen  Scenario,  enemy  evolving  with  us 
(co-evolution) 

-^Competitive  disadvantage 
As  the  need  to  react  to  rapidly  changing 
threats  increases  so  must  our  tactics,  to 
include  design  &  testing  processes 
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uurrent  Design/ 1  esting 

Methods 

DOD  acquisition  system  ill  equipped  to  rapidly 
respond  to  rapidly  morphing  threats,  leading 
to  the  creation  of  new  entities  to  bypass 
existing  acquisitions  processes: 

ACTD  Program 
Rapid  Equipping  Force 
Task  Forces  (IED,  etc.) 


Why  is  this  not  the  norm? 


Large  Acquisitions 

Programs 


Requirements/ 


System 


->  The  Immaculate  Acquisition 


r  asi  auu  me  r  uiiuus 


UAH  Production  Acceleration 


Oct  04 


Jul  04 


May  04 


y  03 


rAOR  Needs  8105*  /  Monthly  Production 


AOR  Needs  5000*  /  Monthly  Production  350 


AOR  Needs  4454  /  Monthly  Production  220 


Jan  04 


AOR  Needs  4149  /  Monthly  /  Production  138 


Oct  03 


AOR  Needs  3279  /  Monthly  Production  81 
First  Acceleration  Dollars  Received 


Aug  03 


AOR  Needs  1407  /  Monthly  Production  80 


AOR  Needs  235  /  Monthly  Production  15 


Deficiencies 

DoD  develops  and  has  paid  for  large  amounts 
of  software  code  that  isn’t  readily  accessible  or 
reusable. 

Interoperability  issues  across  the  services, 
commands  and  systems. 

Services  constantly  reinventing  code 

Increasing  complexity  of  software  code 

Development  costs  outweigh  COTS  costs  (if 
COTS  available) 

Timely  delivery  of  new  solutions 


How  network  centric  systems  are 
acquired  influences  behavior 

Results: 

•  Stove-piped  systems 

•  Inoperable  systems 

•  Slow  creation  of  systems,  lack  of  agility 

•  Less  innovation 

Basic  Premise  of  Solution 

Two  areas  to  change  for  creating  network-centric  systems 

1 .  The  environment  for  how  systems  are  acquired, 
designed,  utilized  and  shared 

2.  Methodologies  for  acquisitions 
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Change  Methods 


Open  Technology  Development  (OTD) 
methodologies  for  hardware  and  software 

DoD  has  spent  huge  amounts  of  money  developing 
software  code,  which  is  rarely  available  for  reuse. 

•  Information  technology  is  the  glue 

•  Open-source  proven  success  in  the  private  sector 

•  Better  systems  components  are  evolved,  evaluated 
and  tested  through  a  distributed  competitive 
collaborative  network. 


OTD  Overview 


Transition  of  publicly  available  OSS  into  (and 
out  of)  DoD 

Development  of  DoD  enterprise  code 
‘repository’*  for  reuse 

Enable  collaboration  across  DoD  on  technology 
acquisition  and  development 


*Not  centralized 
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OTD  Benefits 


•  Speed  of  technology  deployment 

•  Avoid  constant  rebuilding  of  technology 

•  Improve  technological  collaboration 

•  Leverage  external  open  source  technology 
investments 

•  Focus  new  development  in  appropriate  areas 
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OTD  tools 

•  Manage  the  software  development  lifecycle  and 
enable  better  documentation  of  code 

•  More  than  just  a  code  repository  -  community 
and  collaboration  tools 

•  Increased  code  reliability  and  reduction  of 
interoperability  risks 

•  Increased  awareness  about  developed  code. 

•  Potential  savings  though  reuse  of  code 

•  Breeding  ground  for  new  ideas 

•  Treats  code  as  dynamic  and  evolving  vs.  static 
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Industry  Understands 

the  Benefits. 

Corporate  America  is  transitioning 

IBM  -  >  $1B  Investment  in  Open  Source 

Apple  -  OSX  built  on  open  source 

HP  -  over  200  Open  Source  based 
products 

Microsoft  uses  open  source  methods 
internally 

CSC  and  BAE  -  Shifting  to  OSS  Model 


OT&E &  OTD 

•  Testing  &  validation  plays  a  key  role  in  OTD 

•  Community  of  interest  needed  to  rapidly  test  anc 
evaluate  new  systems  and  rapidly  share  test 
technology 

•  Dynamic  environment  needed  to  match  testing 
needs  to  IT  development 

•  OT&E  is  part  of  a  dynamic  environment 

•  Testing  of  NOW  systems  must  move  from  static 
testing  to  constant  dynamic  monitoring. 
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Not  just  Technology 


•  Need  to  focus  on  fostering  the 
creation  of  an  ecosystem  that 
recognizes  (and  rewards)  risk  taking 
and  innovation  coupled  with  open 
architecture  systems. 
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•  Questions? 

•  For  further  information  contact  me  for: 

•  AS&C,  Open  Technology  Design  Report 

•  NUWC  report:  Network-Centric  Warfare,  Total 
Systems  Design  &  Testing,  June  2005. 

•  John  Scott,  johnmscott@mindsprinq.com.  (240)  401.6574 


Effort  initiated  by:  Sue  Payton,  Deputy 
Undersecretary  of  Defense  -  Advanced  Systems  & 
Concepts 
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Open  Source  Software  initiative,  http://www.opensource.org 

Open  Source  Software  vendors:  Collabnet,  http://collab.net  and  SourceForge, 
http://sourceforge.net 

I BM  VC  calls  for  'open'  hardware,  Richard  Goering,  EE  Times,  04/08/2005, 
http://www.eetimes.com/news/design/showArticle.jhtml?articlelD=1 60502705 

Raymond,  E.S.,  The  Cathedral  &  the  Bazaar:  Musings  on  Linux  and  Open  Source  by 
an  Accidental  Revolutionary,  O’Reilly  Publishers,  2001 

Open  Source  Software  Institute  (deal  with  Gov  issues),  www.oss-institute.org 

Open  Source  and  These  United  States  -C.  Justin  Seiferth, 
http://skyscraper.fortunecity.com/mondo/841/documents/99-184.html 


Open  Source  Software  for  Imagery  &  Mapping,  http://www.ossim.org  (great  example) 


Dual  Thrust  Modified  Smokey  Sam 
for  Low  Cost  Testing  and  Simulation 

NDIA  22nd  National  Test  &  Evaluation  Conference 

9  March  2006 


EO  Countermeasures  Technology  Branch 
EO  Sensor  Technology  Division 

Sensor  Directorate 


Air  Force  Research  Laboratory 


Briefer:  Bill  Taylor 
Tech  Advisor  for  EOCM 
AFRL/SNJW 


Project  Description 


*  Objective(s): 

—  Develop  a  low  cost,  dynamic  stimulation  capability  for  EO/IR  missile 
warning  and  directed  IRCM  system 

*  Requirements 

—  Provide  motions  and  nominally  representative  rocket  plume  signature  of 
MAN  PADS  threats 

—  Provide  method  to  stimulate  Close  Loop  Laser  IRCM  testing 

—  Provide  method  to  detect  laser  emissions  from  directed  IRCM  systems 

—  Provide  controlled  trajectory  for  operations  in  close  quarters,  or  heavy  clutter 
areas  as  well  as  free  flight 

*  Technical  Approach: 

—  Use  GTR-18A  SMOKEY  SAM  rocket  as  baseline  capability 
—  Naval  Surface  Warfare  Center  -  Indian  Head  modifies  rocket  motor 
—  Develop  cable  launch  system  for  control  flight 
—  Demonstrate  both  free  &  cable  launch  to  validate  capability 
—  Measures  in-band  intensities  with  reference  to  MANPADS 
—  Conduct  testing  to  demonstrate  capability 


Background 


•  Smokey  Sam  is  well  used  rocket 
for  training  of  aircrews 

•  Testing  using  Smokey  Sam  rockets 
were  useful  in  functional  tests  for 
laboratory  programs 


•  An  improved  capability  was 
thought  to  be  useful  but  needed  to 
be  low  cost  for  S&T  use! 


Generic  IR  SAM  Shot  Signature 


Baseline  missile  launch  and  flyout  case: 

—  Boost  for  1 .5  seconds 

—  Sustain  from  1. 5-7.1  seconds 

—  Burnout  at  7.1  seconds 


Standard  atmospheric  cases  (Desert,  Urban,  Maritime) 


*  Signatures  at  nominal  viewing  angle 

—  Boost:  IR=100W/ster  EO=10mW/ster 

—  Sustain:  IR=10W/ster  EO=3  mW/ster 
—  Burn  Out:  IR=0.1  W/ster  EO=0  mW/ster 


Boost 


Sustain 


Post  Burn  Out  (PBO) 


Rocket  Motor  Modifications 


Propellant  Grain  Modification 

•  The  GTR-18A  Rocket  Motor  was  modified  for  a  two  part  laminated  grain 

•  External  grain  is  standard  zinc  based  fuel 

•  Internal  grain  is  Aluminum  based  grain  used  in  other  operational  system 

•  Combination  gives  a  appearance  Boost  and  Sustain  signature 

•  Thrust  is  low  initial  but  high  for  majority  of  burn  given  improved  trajectory 


iB  o 


Smokey  Sam  Test  Rocket 


•  To  provide  a  reflective  surface 
for  laser  system,  a  nose  cone 
modification  was  developed 
using  coated  bicycle  reflectors 


Trailer  Launch  System 


*  A  telescoping  mast  trailer  was  procured  and  modified 
to  support  cable  launch  operations 

*  2000  ft  steel  .25  in  cable  is  stretched  and  anchored  over 
the  mast 


Launcher 


*  A  low  cost  trolley  was  develop  for  the  cable  system 


Rocket  on  trolley 

Trolley  mounted  on  launcher 


Detectors  System 


Supporting  requirement  to  detect  lasers  emissions 


Detector  and  recorder 
mounted  on  trolley 


Side  view  of  detector/ 
recorder  in  trolley 


T&E  Supported 


*  Laser  Infrared  Flyout  Experiment 

-  2003  Live  Fire  Test  -  WSMR 

—  2004  Live  Fire  Test  -  Tonopah 

*  Affordable  Laser  IRCM  Survivability  Program  -  MWS  testing 
Check  6  Testing 

—  2005  Flight  Testing  -  Aberdeen  Proving  Grounds 

-  2006  Live  Fire  Testing  -  WSMR 

*  Missile  Launch  Detector  Upgrade  Testing  -  WPAFB 

•  2005  UK  Laser  Jammer  Test  -  WPAFB 

•  2006  US  Laser  Jammer  Test  -  WPAFB 

•  Multiple  AFRL  MWS  testing  at  WPAFB 


Development  Process 


*  DT  Smokey  Sam  is  a  supported  element  in  the  overall 
development  process 

—  Concept  Development 

—  Analysis,  and  simulation  studies 

—  Build  prototype  hardware 

—  Early  testing  to  confirm  design  and  simulation  results 

*  Smokey  Sam  firing  provide  low  cost  data 

*  May  need  a  few  real  missile  shots  to  validate 

—  Validate  design  against  simulation  and  empirical  data 


Limitations 


*  DT  Smokey  Sam  has  limitations 

—  Dynamics  after  first  few  seconds  are  not 
representative 

—  Closure  velocity  does  not  present  intensity  rise 
factors  used  by  some  MWS  algorithms 

—  Eject  motor  signature  not  represented  by  rocket 
launch 

*  Working  separate  eject  motor  stimulator  for 
MWS  testing 


Modeling  &  Simulation  Link 


*  Emulation  level  engagement 
modeling  relies  on  empirical 
testing  for  validation 


Accurate  modeling  of  missile 
warning  performance  and 
laser  IRCM  tracking  is  critical 
to  the  design  and  assessment 
of  countermeasures 
effectiveness 


Recap 


Dual  Pulse  Smokey  Sam  (DTSS)  offer  a  low  cost 
capability  for  early  testing,  calibration  of 
optical/infrared  missile  warning  and  Directed  IRCM 
systems 

Modification  of  the  widely  used  GTR-18A  Smokey  Sam 
rocket  with  other  innovation  provides  a  useful  tool  for 
testing  |||| 

As  part  of  concept  development,  DTSS  can  be  used  to 
validate  initial  de^ic|n|  analysis  and  simulations 


•  DTSS  has 


s  like  all  testing  tools 


CAW  use  of  M&S  in  Support  of 
Maritime  HLD/HLS/ATFP  Test 
and  Training  Exercises 


Arman  Tchoubineh 
NDIA  T&E  Conference 
March  6-9,  2006 
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OUTLINE 


*  CAW  Background 

*  Asymmetric  Warfare  Integration  Ctr.  (AWIC) 

*  Why  use  M&S? 

*  Types  of  M&S  used  by  CAW 

*  Summary  and  conclusions 


Storm  ?  Ok  City 

1995 


Munich 

1972 


“We  can  be  certain  about  trends,  but  uncertain  about  events. 
Adapting  to  surprise  quickly  and  decisively  must  therefore 
be  a  condition  of  planning.  Donald  Rumsfeld 
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Lockerbie 

1988 


A  US  Navy  agency  established  in  1999,  dedicated  to 
supporting  US  military  forces  and  local/state/federal 
organizations  in  recognizing,  countering  and  controlling  the 
effects  of  Asymmetric  Warfare  in  support  of  the  Global  War 
on  Terrorism 

Develop  and  execute  complex  scenario-based  Exercises 
involving  multiple  diverse  organizations;  expanding  on 
Maritime/Port  Security  environment  heritage 

-  Testing 


Training 

Experimentation 
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Inside  the  Fence 


^5  Center 

Asymmetric 

Warfare 


Construct  of  the  Experiment 
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Concept 


[ 


GWOT  Mission  Areas 


D 


AT/FP 


MIO 


Raid  Ops 


NEO 


Strike  Ops 


Peace  Ops 


DSCA 


Testing 

Training 

Experimentation 


Instrumented 

Infrastructure 

Live/Virtual/ 
Constructive  1 
Participants  | 

Complex 

Scenarios 

Integrated  GWOT  Environment 

GWOT  Mission  Areas 


Intel  Ops 
Special  Ops 
Nation  Assistance 
Freedom  of 
Navigation 
Counter  Drug  Ops 
Spt  to  Insurgency 
Recovery  Ops 
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Products  and  Services 


Training 

& 

Exercises 


Technology 
Assessment 
&  Analysis 


Integrated  AW  Environment 

Simulation  Development 
Exercise/Research  Support 
Scenario  Dev  &  Documentation 
Terrain  and  Org  Data  Development 
Real/V  irtual  targets/threats 
White  Cell 
Red  Cell  (OPFOR) 

Intel  Cell 
METOC  Cell 


Mission 
Management 
&  Rehearsal 


Tactics  & 
Procedures 
Development 


Metrics 
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Technology  Insertion 


Warfighter 
Higher  Command 


Technology 

Tactics 

Techniques 

Procedures 


Needs 

i 

- 1 

Technical 

Community 


Exercise 
Live  or  Simulated 


Technology 


C n 


Tactics  _  ^ 


War  Fighter 


Academia 


Industry 


INTEL 

Community 


Civil 

Sector 


Tools 


METOC 

GioVIZ,  Keyhole 
E-TEAM/NC4 


Private 

Sector 


DIMAS 


CAW  Exercise 
Control  Center 


JCATS 


RDT&E,  Training,  Systems  Integration 
Joint  Operations 


Asymmetric  Warfare  Integrated 
Center  (AWIC)  Integration 
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Asymmetric  Warfare  Integration 
Center  (AWIC)  Capabilities 


*  Capable  of  distributing  and  recording  data 
(Integrated,  Net-centric) 

*  Live  and  simulated  data  feed 

*  Multi-Level  classified/unclassified  data 

*  Provide  access  to  various  facilities/agencies 

*  Links  to  mobile  data  collection  capabilities 

-Sensors  from  the  CAW  floating  vessel  or  van 

*  Utilized  as  an  Exercise  Control  Center  (ECC) 

*  Utilized  as  a  training  and  communication  center 
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Why  CAW  Uses  M&S? 


Two  aspects  of  the  interoperability  exercise  lend 
themselves  to  simulation:  response  activities  by 
the  participating  agencies  and  Red  operations  (or 
natural  events). 

M&S  is  used  in  our  events  where: 

*  Impact  of  live  play  would  be  prohibitive 

-Obstruction  of/Destruction  to  infrastructure,  Health  hazards 

*  There  are  insufficient  resources 

*  There  is  insufficient  time 

*  A  variety  of  scenarios  and  their  impacts  can  be 
demonstrated 

-  In  an  exercise  or  In  a  Table  Top  discussion 
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Simulation  Characteristics/Benefits 


•  High  Level  Scripting 

-  Loosely  scripted  actions  provide  ground  truth 
representation  that  can  also  be  recorded/played  back 

-Calculated  response  to  commanded  activities 

-  Easily  controlled  ad  hoc  injects 
-Detailed  element  and  terrain  data  required 

•  Scenario  development  and  run  time  costs  are  minor 
therefore  it  reduces  exercise  and  training  costs. 

•  Scenario  plays  out  logically,  with  unrestricted  interactive 
play. 

•  Terrain  and  entity  data  detail  is  readily  available. 
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Info  Activity  Flow 


Live  responder 
executes  in  the  field; 
Reports  to  EOC  and 
Response  Cell  as 
appropriate. 


Live  OPS 


(  Exercise  Control^ 

II  _ i 

V^Center  (ECC)^/ 

Ilk. 

1 

JHOC 

ig 

Response  Cell  “drives” 

2  USCG  Responders 
in  the  JCATS  model. 
Coord  w/ECC.  Reports 
generated  to  IMC  and 
Live  OPS  as  appropriate. 


Incident 

Management 

Center 


(Unified  Command) 


Info  from  USCG  Patrol 
passed  to  SCC-Mobile  by 
satcom  dispatch. 

Coord  w/Response  Cell. 


AGENCY  EOC 

(eg.  SCC-Mobile) 


SIMULATION 


COTP  analyzes 
and  dispatches 
3  vessels  to 
respond  to  the 
situation  (1  Live; 
2  simulated) 


— 
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WSDOT/ 
WSP 
response 
to  BNSF 
incident 


1 


Sultana 

Departs 

POT 


1  I 

0400  0800 


HOPV 

Explosion 


Container 

Ship 

Offloads 

“Hot” 

Containers 


ENDEX 


USCG  MRO 
CASEVAC 

SFD  ICP  establish 
Triage  site,  ground 
CASEVAC 


WSDOT/WSP 
response  to  1-5 
explosion 


r 

1800 

►  CRISIS 


WSHA  near 
overload  request 
FED  support 
thru  State  EOC 


IJ1ARSEC 

TT 

J 

Change 

1 

1 

7  OCT 

/  T 

1 

/  0800 
JBNCE- 

1 - ► 

1200 

WSDOT/WSP/FBI 

coord  to  reopen  1-5 

&  1-90 

WSDOT/WSP 
response  to  1-90 
explosion 


Container  Truck  explodes  1-5  Puyallup  Bridge 


East  Side,  Container 
Truck  explodes  1-90 
Columbia  River  Bridge 


FED  Response,  DCO/DCE  officially 
activated/  FCC  alerted 


SIMULATION 


NOTE:  Times  are  for  planning  purposes  only 
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Warfare  CA  W Modeling  and  Simulation 


•  GeoViz/Keyhole:  Geo-spatial  representation/locator 

•  Tier:  Simulation  tool  to  provide  situational  awareness 

•  RangeView:  M&S  tool  provides  Common  Operating  Picture 


*  Air  Transport  and  Dispersion  (Plume)  Modeling: 

-  HPAC:  Hazardous  Prediction  and  Assessment  Capability  (DTRA) 

-  VLSTRACK:  Vapor,  Liquid  and  Solid  Tracking  (USN) 

-  CAMEO/ALOHA:  Areal  Locations  of  Hazardous  Atmospheres  (NOAA  &  EPA) 

-  CATS/JACE:  Consequences  Assessment  Tool  Set,  Joint  Assessment  of 

Catastrophic  Events  (DTRA,  FEMA  &  NGIC)  15 

- - —  -  ■  -  -  -  . - 
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Decision  Parameters 


*  Live  play  requirements: 

-  If  all  response  play  is  live,  computer  sim  advantage  is  minimal. 

-  If  live  play  is  not  essential,  computer  sim  can  reduce  logistics  and  costs 

*  Scope  and  complexity 

-  As  scope  increases,  computer  sim  becomes  more  effective,  especially 
if  events  are  strongly  linked. 

-  Complex  events  are  easily  managed  within  computer  sim 

*  Preparation  Costs 

-  Late  changes  easily  accommodated  in  computer  sim. 

-  Consider  computer  sim  support  staff  availability/cost  &  prep  time. 

*  Level  of  play 

-  Increase  in  level  causes  exponential  increase  in  C2  and  response 
elements 

*  Participant  availability 

-  Computer  sim  flexibility  for  filling  in  participant  gaps. 

*  Classified  ops 

-  Performed  w /  relative  ease  on  a  secured  computer. 


CA  W  is  providing  training,  technology  testing  and 
evaluation,  and  the  foundation  for  developing  Tactics, 
Techniques,  and  Procedures  (TTP)  for  controlling  and 
neutralizing  the  effects  of  Asymmetric  Warfare  in  complex 
scenario-based  series  of  live/virtual/constructive  exercises. 


Use  of  simulation  has  improved  capabilities  and 
situational  awareness  at  a  reduced  cost. 
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Conclusion 


*  Using  sims  in  conjunction  with  live  play  or  Table 
Top  exercises  can  improve  effectiveness  and 
efficiency  and  reduce  costs  (depending  on  the 
participant’s  interests) 

*  The  live  and  sim  techniques  can  be  combined  to 
make  best  use  of  attributes  from  each. 

*  Together,  they  can  provide  improved  situational 
awareness  for  the  participants. 
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Modeling  and  Simulation  to 
Advance  Test  and  Evaluation 


Presented  to  22nd  Annual  National  Test  and  Evaluation  Conference 


March  6-9,  2006 
Jacksonville,  FL 

Stephen  Attaway 
Sheldon  Tieszen 


Unlimited  release 

Name/Org:  Steven  R.  Heffelfinger,  1532,  Date:  2/1 7/2006 
Guidance  (if  applicable): 


Sandia  is  a  multiprogram  laboratory  operated  by  Sandia  Corporation,  a  Lockheed  Martin  Company, 
for  the  United  States  Department  of  Energy’s  National  Nuclear  Security  Administration 
under  contract  DE-AC04-94AL85000. 
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Acknowledgments 


Authors  are  privileged  to  give  an  overview 
of  the  work  of  many  individuals 

-  Test  and  Evaluation 

•  Experimental  infrastructure  support 

•  Conduct  of  experiments 

-  Modeling  and  Simulation 

•  Numerical  infrastructure  support 

•  Conduct  of  simulations 
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Presentation  Outline 


•  Two  examples  showing  M&S  and  T&E 
synergy  will  be  given 

-  Structural  example  -  Steve  Attaway 

-  Thermal  example  -  Sheldon  Tieszen 

•  General  comments  will  be  given  on  the 
cost  of  a  combined  approach. 
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Synergy 

Both  M&S  and  T&E  have  strengths,  both  have 
weaknesses 

-  Experiment:  The  full  truth  partially  exposed 

•  If  you  cannot  measure  the  key  phenomena,  no  amount  of 
testing  will  give  you  the  understanding  of  margin 

•  Some  things  are  too  costly  to  test  with  existing  technology 

-  Simulation:  The  partial  truth  fully  exposed 

•  If  your  model  does  not  contain  the  key  phenomena,  no  amount 
of  simulation  will  give  you  the  understanding  of  margin 

•  Some  things  are  too  costly  to  model  with  existing  technology 

Combination  of  M&S  and  T&E  is  stronger  than 
individual  components 

-  Information  content  (T&E  strength,  M&S  weakness) 

-  Information  diagnostics  (T&E  weakness,  M&S  strength) 
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Goal  and  Approach 

Our  goal  is  to 

-  Provide  the  technical  basis  for  high-consequence  decisions 
relative  to  system  safety  and  performance  margins 

Our  approach  is  to 

-  Combine  Modeling  and  Simulation  with  Test  and  Evaluation 

-  Use  a  Verification,  Validation  &  Accreditation  framework  to 
establish  that  our  physical  understanding  is  sufficient 

-  Make  risk-informed  decisions  with  respect  to  system  safety 
or  performance  failure  margins 
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M&S  Advances  T&E  to  Decision  Goal 
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Example  #1 
Validation  Test  of  Concrete  Simulations 


r 


Testing  was  used  to  validate 
our  ability  to  simulate  heavy 
damage  to  reinforced  concrete 


How  does  this  test/analysis 
combination  improve  the  prediction  of 
behavior  for  defense  and  homeland 

security  systems? 

•Validated  computational  tools  for 
modeling  collapse  of  concrete 

•Knowledge  gained  from  modeling 
reinforced  concrete  will  benefit  many 
programs 

•Provides  more  accurate  input  to  risk 
based  decisions 


Critical  to  understanding  concrete  behavior 
and  quantifying  design  margins 
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Test  goals 


Capture  the  response  of  a  reinforced  concrete  structure  in  the 
process  of  collapse. 

Quantify  the  load  deflection  curve  defined  in  Uniform  Facility 
Criteria  (UFC  3-340-01) 

Quantify  how  the  membrane  response  contributes  to  the  energy 
absorbing  ability  of  the  structure 

Measure  the  deflections  of  the  wall  as  a  function  of  load 


COMPRESSIVE 

MEMBRANE 


TENSILE 

MEMBRANE 


Figure  10-7.  Typical  Membrane-Enhanced 
Flexural  Resistance  Function. 
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Figure  10-9.  Compressive  Membrane  Action. 


Figure  10-12.  Tensile  Membrane  Action 


ncertainty 
Quantification 


Large  uncertainty  may 
result  in  facilities  being  over 
designed 

Reduction  in  uncertainty 
equals  reduction  in  cost 


Validation  test  was  designed 
to  produce  a  response  in  in 
the  yellow/red  zone 

Current  design  point  is  in  the 
green/yellow  zone. 


Force  Force 


Current  modeling 


Displacement 


Reduction  in  uncertainty 


Surrogate  Load 


A  water  slug  was  used  to  drive 
the  structure  response 


Water  was  slug  accelerated  to 
600.1  ft/sec  on  high  speed  sled 
track  using  48  rocket  motors 


Sandia 

National 

Laboratories 


Test  and  Evaluation  is  Science  in 
Relevant  Conditions 

Perception  of 


Science 
^Community 
Centroid 

Specific 
▲  Focus 


A 

i 

formatio 
equired 
r  decisic 

1 

¥ 

Advanced 

Diagnostics 


All  data 
necessary  to 
quantify 
prediction 
uncertainty! 


Full-Scale  Test  ^ 


Perception  of 
Test  &  Eva). 
Community 


Relevance 


Sandia 

National 

Laboratories 


S  II*  ^ 

SB*1  i  m 

feSto  tm 

■V&ifeZ  JE  - 


Agile  Reaction 
Frame 

Re-usable  post-tension 
reaction  frame  used  to 
confine  soil,  support 
slab,  and  support  wall 
section 


>ries 


Time  lapse  video 
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M&S  Advances  T&E  to  Decision  Goal 
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Science 

Community 
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Lab  Experiments  ) 


Information 
required 
for  decision 


Uncertainty 
uantificatlon- X 


M&S  Desig 
Of  Testin 


All  data 
necessary  to 
quantify 
prediction 
uncertainty! 


Full-Scale  Test  ^ 


Perception  of 
Test  &  Eval- 
Community 
Centriod 


Relevance 
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^Test  Design 
Parameters 

Test  parameters  were  based  on 
statistics-based  multi-dimensional 
sampling 

-  Thousands  of  2D  simulations  were 
used  to  define  a  failure  surface 

-  3D  high  fidelity  simulations  were 
used  to  define  specific  test 
parameters 

Design  parameters: 

-  Load  magnitude 

-  Soil  cover 

-  Soil  strength 

-  Structure  Span 

-  Concrete  strength 


3D  Pre-Test  Prediction 


Eulerian  model  of  water  slug, 
soil,  and  target  structure 

One-way  coupling  between 
Eulerian  model  and  FEM 
model 

FEM  model  included  explicit 
treatment  of  reinforcing 
steel,  post  tensioning, 
gravity  pre-load  and  contact 
between  blocks. 
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est  Results 


1 00%  success  on 
instrumentation 

Pressures  measured  at 
soil/concrete  interface 

Three  independent 
displacement  measurements 

Excellent  high  speed  video 

New  “speckle  correlation” 
displacement  measurement 
technique  provided  full-field 
displacements  and  velocities 
for  back  wall  surface 


Side  view  of  impact 


4x16  ft  water  slug  moving 
at  600  ft/sec 


CTH  simulation 
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damage. 
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Simulation 


Test  Data 


Test  and  Evaluation  is  Science  in 
Relevant  Conditions 


Perception  of 
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Community 
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Specific 
A  Focus 
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Advanced 

Diagnostics 


All  data 
necessary  to 
quantify 
prediction 
uncertainty! 


Full-Scale  Test 


Perception  of 
Test  &  Eva!. 
Community 
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Relevance 


New  diagnostic 
technology  demonstrated 


New  testing  method:  “Speckle 
correlation” 

Provides  3-D  full  field 
displacement  measurement  of 
structure  as  it  collapses 

Pressure  measured  at 
soil/concrete  interface  to 
measure  pressure  in  soil 

Pressures  were  used  to 
validate  soil  model 


Balls  on  simulation  colored 


error 
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ebar  failure  location  and  timing 
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Rebar  failure  location 

•  3D  FEM  models  were  predicting  failure  of  rebar 
in  compression 

•  Many  features  of  the  test  compared  well  with  the 
pre-test  prediction. 


Areas  of  improvement  needed  based  on  comparison 
of  the  pre-test  analysis  with  test  results: 

-  Rebar  failure  needs  improvement 

-  Location  of  rebar  failure  in  pretest  simulations  were 
incorrect 


-  A  fully  coupled  analysis  is  needed  to  correctly 
capture  soil  pressure  as  the  structure  collapses 


Tension  Necking  Failure 


■Structural  Example  Summary 

Testing  challenges 

-  Too  costly  to  do  full-scale  structure  test 

-  Too  costly  to  explore  parameters  space  with  relevant  reduced- 
fidelity  tests 

Simulation  challenges 

-  Too  costly  to  do  full  physics  simulations 

-  Too  costly  to  reduce  uncertainties  without  tests 

Combination  of  M&S  with  T&E  provides  only  realistic 
path  to  solution 

-  Simulation  optimized  the  parameter  space,  designed  the  test, 
evaluated  results 

-  Reduced  uncertainty  in  modeling  by  testing 
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Example  #2 
Weapon  Systems  Qualification 
in  Fire  Environments 


Validation  of  numerical  simulation 

-  Uncertainty  quantification  for  modeling  and  simulation 
requires  validation  data 

Application  of  numerical  simulation 

-  Examples  showing  modeling  and  simulation  greatly 
aids  test  and  evaluation 

General  comments  on  cost 
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Data  Quality/Quantity 
(Measurement  Fidelity  %_gontrol) 


Example  of  validation  for  fire 


Perception  of 
Science 


All  data  are 
necessary  to 
quantify 
prediction 
uncertainty! 


Perception  of 
Test  &  Eval. 
Community 
Centriod 


Relevance 
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prediction  uncertainty  in  plumes 


Helium  plume  -  First  pairwise  momentum/scalar  coupling 

-  2-D  density,  velocity  and  turbulent  statistics. 

Test  Data 


4-Iriuhw  ri.iui  CCTi'^i-wniln*  Erf.  IV....  r^-.. 


3.5 


2.5 


c  1.5 


a5  1 


Fuego  -  2M  node  simulation 


0.5  - 
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■  sample  mean  from  exp. 

-  mean  +/-  confrdence  interval 
♦  TFNS2M  node  grid 

- 

_ i _ 

1 _ L _ J _ 1 

_ 

0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.0 

axial  distance  <rn> 

Developing  quantitative 
comparison  statistics 


Sandia 

National 

Laboratories 


Data  Quality/Quantity 
(Measurement  Fidelity  %_gontrol) 


Example  of  validation  for  fire 


Perception  of 
Science 


All  data  are 
necessary  to 
quantify 
prediction 
uncertainty! 


Perception  of 
Test  &  Eval. 
Community 
Centriod 


Relevance 
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Test  data  establishes 
prediction  uncertainty  in  gas  fires 

Combusting,  non-sooting  &  sooting  gas  fires 

Velocity 


Raw  Data 


1 

J 


51 


Methane  Fire 
1  m  diameter 
1  M  nodes 


Time  =  35.75  sec 

T  (K) 
2100 1 — - 
1900 1 
1700 1 

1500  r 

1300 
1100 
900 
700 
500 
300 1—1 


Turbulent  Statistics 


Fuego  Simulation 


Data  Quality/Quantity 
(Measurement  Fidelity  %_gontrol) 


Example  of  validation  for  fire 


Perception  of 
Science 


All  data  are 
necessary  to 
quantify 
prediction 
uncertainty! 


Perception  of 
Test  &  Eval. 
Community 
Centriod 


Relevance 
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Soot  Concentration  (ppm) 


Test  data  establishes  - 
rediction  uncertainty  in  quiescent  pool  fires 


BVG-65K  Mesh 
- BVG-250K  Mesh 

-  -  ■  BVG-500K  Mesh 

- BVG-1000K  Mesh 

- TFNS-250K  Mesh 

-  -  'TFNS-500K  Mesh 
- TFNS-1000K  Mesh 

□  Experimental  Data 


Radius  at  1  m  elevation 


Soot  Concentration 


BVG-65K  Mesh 
- BVG-250K  Mesh 

-  -  -  BVG-500K  Mesh 

- BVG-1000K  Mesh 

- TFNS-250K  Mesh 

-  -  -TFNS-500K  Mesh 
- TFNS-1000K  Mesh 

□  Experimental  Data 


Elevation  (cm) 

Centerline 


•  JP-8  2m  Dia. 
quiescent  Fire 

•  Soot 

concentration,  a 
modeled  quantity, 
is  not  accurately 
predicted. 


1.2  M  node  Fuego  sim. 
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Data  Quality/Quantity 
(Measurement  Fidelity  %_gontrol) 


Example  of  validation  for  fire 


Perception  of 
Science 


All  data  are 
necessary  to 
quantify 
prediction 
uncertainty! 


Perception  of 
Test  &  Eval. 
Community 
Centriod 


Relevance 
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Test  data  establishes  - 

prediction  uncertainty  in  cross  wind  pool  fires 

JP-8  2m  Dia.  Cross-Wind  Fire 
Waterloo.  Canada  test  facility 


_ _! 


Plenum 


Pool 


C/MF 


Test  Section 


Exit  with  Obstruction 
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The  Challenge  of  Non-linear  Physics 
Prediction  in  Relevant  Geometries 


•  Balls  are  experimental  data,  colored  with  temperature. 

•  Surfaces  are  simulation,  colored  with  temperature. 

•  Temperature  scales  are  the  same  for  both. 

One  is 

thermal  response 

is  well  predicted,  ^  \  *lpg|p 

the  second  is  not. 


Predicting  thermal 
response  from 
fire. 


Sensitivity  studies 
show  that  fire  is 
predicted 
strongest  on  the 
centerline  while 
test  is  shifted 
right. 


$imT 


vL 


Time  =  0.00 


1073 

993 

913 

833 

753 

673 

593 

513 

433 

353 

273 


In  this  and  every  example  we  have  run  of  relevant  test  problems,  the  results 
have  given  us  insight  into  parameter  sensitivities.  Quantifying  uncertainty  in 
relevant  geometries  with  relevant  physics  is  challenging. 


88SSSS8883S 


Data  Quality/Quantity 
(Measurement  Fidelity  %_gontrol) 


M&S  Advances  T&E  to  Decision  Goal 


Perception  of 
Science 
Community 
Centroid 

A 


All  data  are 
necessary  to 
quantify 
prediction 
uncertainty! 


Perception  of 
Test  &  Eval. 
Community 
Centriod 


Relevance 
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odeling  &  Simulation  in  Support 
of  Weapon  System  Qualification 

Challenges 

-Infinite  number  of  accident/incident 
environments 

•  Large  parameter  space 

-Complex  systems  have  multiple  response 
modes 

•  Different  loads  challenge  different  modes 

-Different  safety  &  security  philosophies 

•  Different  levels  of  acceptable  outcome 
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Modeling  and  Simulation  of  Large 
Parameter  Spaces 


-  M&S  strength:  Hundreds  to  thousands  of  low 
resolution  simulations  can  conducted 


Case  # 

Hole  Size 
(m2) 

Fuel  Pool 
Location 
(m) 

Object 

Enclosure 

Height 

(m) 

Pool  Size 
(m) 

#  of 
Holes 

Wind 

Speed 

(m/s) 

Peak  Heat 
Flux  (kW/m2) 

i 

0.47 

0 

No 

1 

10 

i 

2-10 

99 

2 

5.57 

0 

No 

1 

10 

i 

2-10 

175 

3 

3.0 

1.85 

No 

1 

10 

i 

2-10 

235 

4 

0.47 

3.7 

No 

1 

10 

i 

2-10 

80 

5 

5.57 

3.7 

No 

1 

10 

i 

2-10 

247 

6 

3.0 

1.85 

Yes 

1 

10 

i 

2-10 

175 

7 

0.47 

3.7 

Yes 

1 

10 

i 

2-10 

72 

8 

5.57 

3.7 

Yes 

1 

10 

i 

2-10 

255 

9 

5.57 

Wind  Side 

No 

0 

10 

i 

2-10 

34 

10 

5.57 

Wind  Side 

No 

1 

10 

2 

2-10 

200 

11a 

(Case  5) 

5.57 

3.7 

No 

1 

5 

1 

2-10 

133 

11b 

(Case  13) 

5.57 

Wind  Side 
(2m  up) 

No 

0 

5 

2 

2-10 

266 

12 

5.57 

Wind  Side 
(2m  up) 

No 

1 

10 

2 

2-10 

171 

13 

5.57 

Wind  Side 
(2m  up) 

No 

0 

10 

2 

2-10 

254 

Temperature  [PC) 


?1hilh:  Y=39  95 


Max 


3001.1 
20QO.Q 
IftOO .  0 
1500.0 
1 500.0 
1300.0 
1200.0 
1150.0 
1100.  .0 
900.0 
750.0 
son.  fi 

390.0 

393.0 
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Testing  the  Selected  Scenario 


Test  results  showed 

-  Thermal  loads  for  specified 
scenario 

-  Steady  simulations 
predicted  peak  heat  flux 
levels  but  average  levels 
were  ~  50%  lower  than 
prediction 

-  In  spite  of  our  best  efforts 
to  control  boundary 
conditions  outdoors,  wind 
effects  too  much  to 
reproduce  test  for 
expensive  hardware 

-  We  have  very  recently 
moved  indoors 
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esigning  a  Wind  Tunnel  for  Fires 
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•  Desire  to  reproduce  outdoor  fires  indoors  to 
control  boundary  conditions. 

•  Simulations  used  to  design  and  commission 
cross-wind  test  facility 


Cross  Wind 
Fire  Test 
Facility 


Designing  the  Indoor  Test 

Design  with  M&S  to  reproduce  the  heat  flux 
levels  in  the  outdoor  fire. 

Current  status:  Design  is  locked  and  testing  is 
imminent. 
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Fire  Example  Summary 

Testing  challenges 

-  Too  costly  to  explore  parameters  space  with  relevant  reduced- 
fidelity  tests 

-  Full-scale  structure  tests  are  too  costly  to  design  with  trial  and 
error  empirical  learning  approaches 

Simulation  challenges 

-  Tests  are  required  for  validation  to  establish  uncertainties 

-  Uncertainty  for  complex  non-linear  physics  problems  in  real 
geometries  is  not  easy  to  establish 

Combination  of  M&S  with  T&E  provides  only  realistic 

path  to  solution 

-  Simulation  optimized  the  parameter  space  and  designed  the 
tests 

-  Reduced  uncertainty  in  modeling  by  testing 
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eneral  Comments  on  Cost 


Which  is  cheaper  Modeling 
and  Simulation  or  Test  and 
Evaluation? 

-  Both  have  infrastructure 
costs,  both  have  use  costs 

•  In  general  infrastructure  costs 
are  high  relative  to  use  costs 


O" 

0 

CD 

■Q 

0 


No  Yes 


Infrastructure  in  place? 


-  Either  approach  can  be  cheaper 

•  Do  you  have  the  requisite  infrastructure  in  place? 

-  Relatively  easy  to  answer 

•  Can  you  extract  the  key  knowledge? 

-  Relatively  hard  to  answer 
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General 


Comments  on  Cost 


Is  the  combination  of  M&S  and  T&E  cheaper? 

-  In  general,  no. 

•  Infrastructure  costs  are  high  and  not  shared  so  cost  effectively 
doubles  relative  to  the  cost  if  the  key  knowledge  can  be  obtained  by 
single  approach 

-  Can  the  combination  be  cheaper?  Yes 

•  With  an  unknown  unknown,  what  approach  will  yield  the  necessary 
knowledge? 

•  Dual  track  approaches  are  commonly  used  to  minimize  risk  when  a 
given  approach  can  fail,  but  the  combination  is  likely  to  succeed 

-  Optimum  dual  track  approach  is  one  which  minimizes  common  mode 
failure  and  maximizes  synergy 

•  M&S  in  combination  with  T&E  has  highest  likelihood  to  succeed  for 
supporting  high-consequence  decisions  involving  system  margins 

-  The  authors  contend  that  neither  example  given  would  have  been 
successful  with  the  combination  of  modeling  and  simulation 
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ummary  Experience  to  Date 


•  The  combination  of  M&S  and  T&E  is  analogous  to  the  well 
established  scientific  method. 

-  M&S  is  the  current  codification  of  theory. 

-  T&E  is  experimentation  under  realistic  conditions. 

•  The  goal  is  to  have  sufficient  understanding  of  the  truth  to 
make  high-consequence  decisions  with  respect  to  failure 
margins. 

-  Our  experience  is  that  the  M&S/T&E  combination  has  yielded  much 
more  confidence  in  our  decisions  that  would  be  obtained  by  either 
approach  alone. 

-  The  approach  in  the  examples  is  generic  and  broadly  applicable. 

•  Cost/benefit  demonstration  for  dual  track  approaches  will 
remain  challenging 

-  Difficult  to  establish  savings  -  combined  costs  appear  high  if  failures 
are  avoided  -  hard  to  prove  we  avoided  failure  by  this  approach. 
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LFT&E 


An  LFT&E  Perspective  on  Making 
M&S  and  T&E  Better  Partners 


Dr.  Lowell  Tonnessen 
Institute  for  Defense  Analyses 


T  onnessen-3/1 4/2006-1 


Presented  at  NDIA  Test  &  Evaluation  Conference 

Jacksonville,  Florida 
6-9  March  2006 


DOT&E  Assessments 


— - " 

•  Operational  Effectiveness 

•  Operational  Suitability 

•  Survivability 


Lethality  --  weapons  systems 
and  munitions 

Vulnerability  --manned  systems 


What  Makes  LFT&E  Different? 


•  Operational  Tests 

-Force-on-force 


-Systems  operated  by  typical  users 
-Users  protected  from  live  fire 
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Legislative  Emphasis  on 
User  Casualties 


The  term  ‘realistic  survivability  testing’ 
means... 

testing  for  vulnerability  of  the  system  in 
combat  by  firing  munitions  likely  to  be 
encountered  in  combat... 

with  the  primary  emphasis  on  testing 
vulnerability  with  respect  to  potential 
user  casualties.... 


10  use  2366 
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Basic  Principles 


—  LFT&E 


•  It  is  possible  to  craft  an  LFT&E 
program  that  does  not  include 
modeling  and  simulation,  but 
probably  not  wise 

•  M&S  is  a  tool  in  support  of  both 
test  and  evaluation,  not  an 
alternative  to  testing 
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CO  QO 


LFT&E 


Test  & 


•  Shot  selection 

•  Test  sequencing 

•  Pre-test  prediction 
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Evaluation 


•  Integration  of  test 
results 

•  Extension  to  untested 
conditions 

•  System-level  analyses 

•  Validation  &  model 
improvement 


Vulnerability  and  Lethality 


—  LFT&E 


•  Unexpected  vulnerabilities, 
personnel  casualties 

•  M&S  tends  to  underestimate 
lethality,  overestimate  survivability 

•  Operational  lethality  vs.  parametric 
lethality 
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Validity 


—  LFT&E 


We  test  in  order  to  validate  system 
performance.  We  simulate 
because  we  cannot  afford  to  test 
everywhere 

"If  we  don't  test,  the  model  is 

always  right."  (Jim  O'Bryon) 
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F-22  Wing  Section  Test 
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Benefits  of  M&S? 

— - 

"Modeling  and  simulation  offers  the 
F-22  program  another  benefit/'  Air 
Force  sources  said.  "Because  the 
Service  would  control  the  inputs  into 
the  model,  the  outcome  —  proving 
the  aircraft's  effectiveness  —  is 
much  easier  to  shape  than  the 
outcome  of  an  open  air  test  with  any 
number  of  unanticipated  variables." 

Inside  the  Pentagon,  September  1,  1995 
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Congressional  Perspective 


—  LFT&E 


•  All  sources  of  data  are  not 
equal 

•  A  number  of  laws  require 
testing;  few  laws  require  M&S 
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Learning  and  Evaluation 


—  LFT&E 


•  M&S  is  of  undoubted  benefit  in 
designing  more  survivable,  or  more 
lethal,  systems 
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Learning  and  Evaluation 


•  Much  can  be  learned  from  a  single 
realistic  event 
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Learning  and  Evaluation 


•  Much  can  be  learned  from  a  single 
realistic  event 
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■ 

[  Learning  and  Evaluation 

■ 

•  Much  can  be  learned  from  a  single 
realistic  event 
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Learning  and  Evaluation 


•  It  is  misleading  to  compare  relative 
numbers  of  test  shots  and 
simulated  shots 

•  Beware  the  appeal  of  “statistically 
significant”  sample  sizes  based  on 
M&S 

•  “Does  the  model  give  1000  shots  worth 
of  information,  or  the  information  from 
one  shot  1000  times  over?" 
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Learning  and  Evaluation 


Live  Fire  Testing  is  limited  to  a  small 
number  of  tests,  but  each  one  provides 
the  opportunity  for  surprises 

Surprises  occur  regularly 


•  The  level  of  test  realism  determines  the 
kinds  of  surprises  that  are  possible 
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Learning  and  Evaluation 


•  If  M&S  results  are  to  affect  design,  the 
information  must  be  credible  at  the  component 
level 

•  OSD’s  M1A1  Abrams  Tank  LFT&E  report  noted 
that  the  model  “...in  its  current  state,  would  be 
inadequate  for  evaluating  the  effectiveness 
reduction  of  design  changes.  Over  half  of  the 
critical  components  damaged  leading  to  a  loss 
of  function  were  either  not  predicted  ...  or  were 
assessed  as  rare  events....” 
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Pretest  Predictions 


—  LFT&E 


•  “We'll  know  we’re  successful  in  live 
fire  testing  when  the  modeling  tools 
we  use  are  so  successful  that  there 
are  no  surprises  in  LFT."  ( Gen  Larry 
Welch) 

•  Surprises  are  only  meaningful  in 
terms  of  expectations,  and 
expectations  are  defined  by 
pretest  predictions. 
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Pretest  Predictions 

— - “ 

•  Pretest  predictions  do  not  have  to 
employ  computer  models 

•  Predictions  should  contribute  to  the 
planned  evaluation 

•  The  prediction  should  be  capable  of 
being  compared  with  the  test  damage 
assessment 

"In  assessing  the  value  of  computer  models,  the 
"golden  rule"  applies:  a  model  must  predict  with 
higher  accuracy  than  the  consensus  process 

(engineering  judgment)." 
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Challenges  --  Personnel  Casualties 

— - 2 - - 

•  Assessment  of  personnel  casualties  is 
increasingly  relevant,  and  presents  a 
major  challenge  for  M&S 

•  Personnel  casualties  are  important  in 
their  own  right,  and  not  just  as  a 
contributor  to  mission  kill 

•  If  we  expect  to  make  progress  towards 
reducing  casualties,  we  need  a  means 
to  track  that  progress 
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Challenges  --  Ships 

Ship  programs  are  looking  for 
alternatives  to  full  ship  shock  trials 


•  M&S  will  need  to  be  developed  to 
support  these  alternatives 
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Other  M&S  Challenges 


—  LFT&E 


•  Proprietary  M&S 

•  Integrated  survivability  M&S 
(detectability,  hit  avoidance, 
vulnerability,  recoverability) 

•  Timely  correction  of  M&S-based 
estimates  to  account  for  test  results 

•  Ballistic  M&S 
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Conclusion 

— ’ 

•  LFT&E  needs  M&S  to  properly  evaluate 
system  vulnerability  or  lethality 

•  Improvement  is  needed  in  several  areas 

•  These  include  the  assessment  of 
personnel  casualties,  and  the  integration 
of  vulnerability  M&S  into  an  integrated 
survivability  assessment 
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^ERC/Af. 


CAS-TAS 


at  ions 


Mr.  Stan  Ulkoski 

J.  F.  Taylor,  Inc. 

( contractor  support) 

PMA-207D2 

Commercial  Air  Services 
Tactical  Air  Services 


CAS  /  TAS  Organization 


WAYNE  FILARDO 

MIKE  MORRIS 

RDT&E  Sr  Lead  Engineer 

Scheduling 

J.F.Taylor,  Inc. 

Wyle  Labs 

301-862-3941 

301-757-9571 

ELLEN  HOLCOMB 

LINDA  BARRETT 

Finance/Scheduling 

Admin/Fuel 

Scheduling 

J.F.Taylor,  Inc. 

Wyle  Labs 

301-862-3939 

301  -757-9572 

HARVEY  ARMSTRONG 

STAN  ULKOSKI 

Logistics 

Aircraft  Ops  Mgr 

J.F.Taylor,  Inc. 

J.F.Taylor,  Inc 

301-757-9529 

301-862-3939 
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•Fleet  support/DOD  -  Provide  cost  effective  capable  aircraft  utilizing 
commercial,  RDT&E  prototype  and  generic  EW  systems  to  meet 
training  requirements  with  no  degradation  to  training. 


•RDT&E  -  Provide  to  the  DOD  community  cost  effective  aircraft  that  can 
be  modified  to  meet  advance  system  development  requirements  and 
access  to  DOD  airborne  training  systems. 


National  Defense 


•PMA-207D2  front  line  training  support. 

•Prior  to  fleet  deployment  Commercial  Air  contractors  provide  OPFOR 
and  ADFOR. 

•Missions  flown  by  C/TAS  provide  the  expected  hostile  environment  the 
battle  group  will  encounter. 


opposing  forces 


friendly  forces 


1  aircraft 

12  aircraft 
8 


ircraft 


Prop 
Lear  Jet 
Alpha  Jet  (4) 

Hawker  Hunter  (4) 

Kfir  6  aircraft 

G-1  Gulfstream  2  aircraft 
Tanker  1  aircraft 


Total  CAS/TAS  Fleet  support  aircraft 


30  Aircraft 


Flight  International,  Inc 

Newport  News,  VA 
North  Island,  San  Diego,  CA 

Learjet  and  Cheyenne 


Airborne  Tactical  Advantage  Company 
Newport  News,  VA 

F-21C2  Kfir/Hawker  Hunter 


Phoenix  Air,  Inc 

Cartersville,  GA 

G-1  Gulfstream 


Air  USA,  Inc 

Quincy,  IL 

Alpha  Jet 


Omega  Aerial  Refueling  Services,  Inc 

Alexandria,  VA  -==  n 


KC707  Tanker 


OMEGA 
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WHY  COMMERCIAL  AIR  SERVICES??? 


1. 

2. 

3. 

4. 

5. 

6. 

7. 

8. 

9. 

10. 


Over  20  years  of  RDT&E  experience,  working  with  FAA,  contractors,  and  DOD  agencies 

Central  point  for  air  assets 

Cost  effective  aircraft  and  EW  support  systems 

97%  Mission  completion  rate 

Contract  in  place 

Small  foot  print 

Majority  of  the  pilots  are  former  military 
Engineering  in  house 
Global  support 
Financial  tracking  provided 


1. 

2. 

3. 

4. 

5. 

6. 

7. 

8. 

9. 

10. 
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WHY  COMMERCIAL  AIR  SERVICES??? 


•  OPSEC  capable 

•  In-house  or  can  access:  EA  /  ECM  /  threat  sim  /  chaff  pods,  tow  targets,  and  prototype  EW 
systems 

14.  Extensive  electronic  warefare  experience  in  military  training,  RDT&E  and  contractor  RDT&E 

15.  FAA  certified  (STC,  DER,  Form-337  capable) 

16.  Available  to  government  agencies  and  civilian  contractors  supporting  government  projects 


WORLD-WIDE  SUPPORT 

DOD  and  RDT&E 


www.electwarairc.com 


T&E  Transformation 

We  Don't  Need  a  More  Formal  Invitation 


William  P.  Yeakel 

wpy@orsacorp.com 
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“Far  more  than  just  materiel  solutions,  Army 
Transformation  is  a  multidimensional  process  which 
involves  adapting  new  technologies  to  war  fighting; 
developing  improved  Joint  Operating  Concepts  and 
business  processes;  changing  organizational  structures,  and 
developing  leaders,  people,  and  a  culture  that  are  relevant  to 
the  future.  It's  important  to  remember  that  Army 
Transformation  is  not  an  end  state;  it  is  a  continuous 
journey  in  which  the  Operational  Army  and  the  Institutional 
Army  adapt  to  an  ever-changing  operational  environment. 
As  such,  Army  Transformation  is  ingrained  in  everything 
we  do  -  today,  tomorrow  and  forever.” 


Francis  Harvey,  SEC  ARMY 


“We  recognized  that  to  put  weapon  and  system 
capabilities  quickly  into  the  hands  of  the  warfighters, 
we  could  not  conduct  business  as  usual.” 

MG  James  R.  Myles,  CG  ATEC 


•  Global  War  on  Terror 

•  Tightened  schedules 

•  Greater  competition  for  resources 

•  Emphasis  on  jointness 

•  Facilities,  equipment  &  personnel  under 
stress 

•  Changing  players,  roles  &  responsibilities 


Current  Situation 


What  About  M&S 


Delivery  consistently  falls  short  of  promise! 

T&E  community  faced  with  greater  risk! 

T&E  community  largely  unfamiliar  with  Army  and 
Joint  models/simulations! 

Expensive,  not  always  cost  effective! 


Causes 


Model  developers  have  different  objectives 

Greater  M&S  expertise  needed  in  T&E  community 

Lack  of  real  M&S  understanding  at  senior  levels 
throughout  DOD 

T&E  is  often  an  after-thought  for  M&S  world 


Recommendations 


•  Invest  in  M&S  education 

-  Learn  how  and  when  to  use  M&S 

-  Learn  when  not  to  and  why 

-  Understand  data  requirements/implications 

-  Know  the  language 


Put  T&E  requirements  on  M&S  developer's  table  early 
in  process 


•  Focus  M&S  on  managing  T&E  risk 
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More  Recommendations 


Employ  a  consistent  framework  in  T&E  approach. 

-  MMF  or  something  like  it  is  essential  particularly  for  Systems 
of  Systems 

-  Complexity  and  interdependence  of  issues  too  great 
otherwise 


Listen  to  Army  CoS  and  SECARMY 

-  Ask  yourself  and  your  boss  how  you  can  support  the  soldier 
more  effectively 

-  Learn  about  and  use  Lean  Six  Sigma  approach  and  force 
M&S  to  support  efficiency  without  sacrificing  effectiveness 


Thank  You! 


Finally,  despite  the  fact  that  what  I  have  said 
may  focus  on  areas  where  we  need  to 
improve,  I  believe  that  you  are  providing 
tremendous  support  for  the  men  and  women 
serving  our  country  in  uniform. 

Thank  you  for  what  you  do  and  the  passion 
with  which  you  do  it! 


